Measuring AI usage without confusing activity with capability

Define the measurement decision

Choose a decision such as whether to improve access, revise enablement, investigate a workflow, change review capacity, or continue a pilot. Define the population, period, approved tools, eligible tasks, unit of activity, and exclusions. A rate without an eligible denominator can misrepresent both adoption and non-use.

Record what telemetry can and cannot show. Product events may demonstrate that a feature was invoked, but not the complete task, checks performed elsewhere, source authority, or the user's reasoning. Content visibility also depends on provider, plan, permissions, configuration, and policy.

The NIST AI RMF Core asks organizations to document context, intended purpose, affected actors, human oversight, measurement, feedback, and monitoring. It supports connecting activity to a defined use rather than treating volume as a standalone outcome.

Sources: [1]

Build the usage-to-work evidence ladder

Create one layer for tool availability and permission, one for eligible users and tasks, one for access, one for interaction events, one for task completion, one for observable process behavior, one for output quality, and one for downstream consequence. For each layer, record the source, denominator, missingness, owner, time window, confidence, and decision it can support.

Do not force every layer into a single score. High access with low use may indicate poor task fit, limited time, policy uncertainty, accessibility barriers, or inadequate support. High use with weak output evidence may require deeper workflow review rather than more promotion.

Define how the measures relate without claiming causation. A sequence from access to interaction to observed work can support investigation, but each link needs evidence. Changes in workload, staffing, incentives, data quality, or reviewer capacity may explain the pattern.

The OECD's 2026 Building an AI-ready public workforce describes workforce capability, governance, strategic alignment, experimentation, and continuous learning as connected conditions. Its public-sector evidence informs interpretation but does not define a universal usage target.

Sources: [2]

Worked artifact: a synthetic policy-research team

Imagine a synthetic policy team whose dashboard shows frequent use of an approved assistant. The ladder confirms that most analysts have access and that summarization is an eligible task. Interaction counts are high, but no direct link exists between a prompt event and a released policy note.

A controlled review samples ten synthetic research tasks. Several analysts use the assistant to organize sources, while others draft manually. One high-use case cites an obsolete consultation document; one low-use case accurately reconciles the final regulation and supporting guidance. Usage alone does not distinguish the two outcomes.

The programme owner adds a bounded evidence review that records source selection, material corrections, final artifacts, and missing telemetry for representative tasks. The next decision is to improve source-version cues and manager guidance, not rank people by prompt count.

The sample is synthetic and cannot establish organization-wide impact, causal improvement, or individual capability outside the observed work.

Verify the ladder and preserve privacy boundaries

Reconcile dashboard definitions with event schemas, permissions, retention, bot or test traffic, duplicate sessions, and actual eligible populations. Sample tasks from different roles and include null and adverse findings. Review whether the data collection is proportionate and whether affected people understand its purpose and limits.

Reopen the ladder when tools, telemetry, workflows, populations, policies, or decisions change. Two Prune can provide controlled task evidence for human interpretation; it does not operate employee surveillance, infer capability from raw activity, or automate employment decisions.