Practical guide

Common mistakes when measuring AI adoption

AI adoption dashboards often fail before the chart is drawn. They count access as use, choose a denominator from licences rather than work opportunities, merge incompatible events, hide missing records, or imply that observed activity caused an outcome. A useful audit tests whether the metric supports its named decision and withholds it when the evidence cannot carry that interpretation. Use the.

By Two Prune

AI adoption dashboards often fail before the chart is drawn. They count access as use, choose a denominator from licences rather than work opportunities, merge incompatible events, hide missing records, or imply that observed activity caused an outcome. A useful audit tests whether the metric supports its named decision and withholds it when the evidence cannot carry that interpretation. Use the TRACE audit: Test the construct, Rebuild the denominator, Audit events and joins, Check rival explanations, and Explain limits. TRACE is a release check for an existing measure, not a substitute for designing a complete measurement system.

Mistake 1: the label outruns the construct

A metric named “AI adoption” may count provisioned accounts, logins, prompts, enabled features, completed workflows, repeated use, reviewed artifacts, or outcome changes. These are different constructs. Start the audit by copying the metric name, decision, numerator, denominator, window, and data source into a measurement card. Then state the observable behaviour in plain language.

Test whether the numerator actually represents that behaviour. A login shows access and activity, not that an eligible task occurred, an output was used, or required verification happened. A survey response can reveal experience and barriers, but it does not independently establish workflow behaviour. A realistic assessment can show capability under controlled conditions, but it does not establish routine workplace use.

The NIST AI Risk Management Framework Measure playbook recommends context-relevant metrics, documentation of unmeasured risks, histories or audit logs, and measurement of human oversight, downstream actions, errors, complaints, and responses. That breadth is a warning against making one convenient event represent the whole construct. NIST AI RMF Measure playbook

Sources: [1]

Mistake 2: the denominator is convenient rather than eligible

Licence counts are easy to retrieve, but they may include service accounts, leavers, people without a relevant task, people on leave, and roles excluded from the workflow. A headcount denominator can also be wrong when opportunities vary sharply by person or period.

Rebuild the denominator from the decision. For a sustained-use question, record eligible people and the minimum opportunity needed to exhibit sustained behaviour. For a workflow-coverage question, count eligible cases or tasks. Show people without an opportunity separately from people who had an opportunity and did not use the workflow.

Run sensitivity checks. If the result changes materially when short-tenure staff, low-opportunity roles, or migrated records are handled differently, publish the range and assumptions rather than selecting the most flattering version.

Mistake 3: event semantics, identity, and joins drift

An event name is not a stable definition. A product release may fire the same event at a different stage; retries may create duplicates; batch processes and service accounts may resemble human activity; identifiers may change during a migration; time zones may move events across reporting windows.

Audit the lineage from source event to headline number. Sample raw records and verify the actor, workflow, event meaning, timestamp, version, deduplication rule, and link to an eligible opportunity. Reconcile counts before and after every join. Track unmatched records by source, role, team, tool version, and date.

The UK Government Data Quality Framework treats quality as fitness for purpose across the data lifecycle and emphasizes clear documentation, metadata, known issues, and preparation for system change. Apply those controls to telemetry definitions and transformations, not only to the final dataset. Government Data Quality Framework

Sources: [2]

Mistake 4: missingness is converted into zero or silently dropped

Missing evidence can mean many things: no opportunity, no use, failed instrumentation, blocked linkage, privacy-preserving aggregation, or an incomplete source. Replacing every missing value with non-use depresses the measure; dropping every missing record can inflate it. Both choices conceal uncertainty.

Create a missingness ledger with reason, count, affected group, likely direction of bias, repair owner, and release disposition. Choose among repair and rerun, explicit unknown, bounded estimate with sensitivity analysis, publish with limitation, or withhold. Do not impute a value merely because the dashboard schema requires one.

Check whether missingness is patterned. If a tool migration affects one region or a workflow-system feed excludes escalated cases, the remaining data may not represent the population relevant to the decision.

Mistake 5: activity is treated as quality, impact, or causation

More events may reflect more eligible work, automated retries, a mandatory training exercise, changed instrumentation, manager pressure, or novelty. Even correctly measured sustained use does not establish capable use, control effectiveness, productivity, service quality, or causal impact.

List rival explanations before interpreting a trend. Compare source definitions and opportunity volume across periods. When a business outcome is in scope, specify a theory of change and an evaluation design capable of distinguishing the intervention from other changes.

The UK guidance on impact evaluation of AI interventions recommends early evaluation planning, a fully specified theory of change, appropriate comparison, attention to changing AI systems, and transparency about what can be learned. A descriptive adoption series should stay descriptive unless the design supports a stronger conclusion. Guidance on the impact evaluation of AI interventions

Sources: [3]

Worked artifact: repairing a finance adoption dashboard

This synthetic audit concerns an AI-assisted invoice-exception workflow in a 160-person finance shared-services group. The dashboard reports 91% adoption because 146 licensed accounts logged in during the month.

TRACE finds that the label cannot be supported:

| Audit step | Dashboard input | Finding | Correct disposition | | --- | --- | --- | --- | | Test construct | Any monthly login | Shows access or activity, not invoice-exception workflow use | Rename the access metric and build a workflow measure | | Rebuild denominator | 160 employees | 140 people were eligible; 12 roles had no invoice task and 8 people were absent for the full window | Use 140 for a person-based opportunity view | | Audit events | 146 logins | 11 were service or test accounts, 18 were duplicated retry events, and logins did not link to invoice cases | Exclude non-human and duplicate records; do not infer workflow use | | Audit joins | Tool identifier to case system | A migration left 96 of 1,280 eligible cases without the new actor key | Repair linkage or report those cases as unknown | | Check interpretation | “Adoption increased after training” | Training, a quarter-end volume spike, and the identity migration occurred in the same period | Do not attribute the change to training |

After lineage repair, the team defines sustained workflow use as an eligible person completing at least four linked invoice-exception cases across two separate weeks, with the required reviewer disposition preserved. Eighty-three of 140 eligible people meet that definition. Opportunity visibility is 1,184 of 1,280 cases after repair; the remaining 96 cases stay unknown rather than being counted as non-use.

A risk-selected artifact review finds that 38 of 44 sampled cases contain the required evidence and disposition. That sample is reported separately; it is not multiplied by the sustained-use proportion to manufacture a “quality-adjusted adoption rate.” The release packet includes definitions, source versions, join success, exclusions, unknowns, sensitivity results, reviewer findings, and a change log.

The final dashboard publishes access, sustained workflow use, opportunity visibility, and sampled review findings as separate measures. It explicitly states that the series does not establish productivity improvement, causal impact, or capability among people who lacked an eligible opportunity.

A release checklist for adoption measures

Before publishing, confirm that:

  • the named decision and observable construct match;
  • numerator and denominator use the same population, workflow, and window;
  • eligibility and opportunity rules are recorded before the result is interpreted;
  • event semantics, versions, duplicates, service activity, identities, and joins are tested;
  • missing records have a visible disposition and sensitivity check;
  • quality and control evidence remain distinct from activity;
  • rival explanations and system changes are documented;
  • causal language appears only when the evaluation supports it;
  • privacy, access, retention, and aggregation choices are proportionate to the decision;
  • the metric has an owner, refresh date, revision log, and publish, repair, hold, or withhold decision.

Two Prune boundary

Two Prune can assess observed performance in realistic simulations and explain the evidence behind an assessment score. That controlled capability evidence can inform a broader adoption evaluation, but it does not measure organisation-wide workplace use, infer why adoption changed, monitor every workflow, or make employment decisions. Customers remain responsible for telemetry governance, denominators, data quality, evaluation design, privacy, and operational interpretation.

Sources

  1. 1.NIST AI RMF Measure playbook · National Institute of Standards and Technology
  2. 2.Government Data Quality Framework · UK Government
  3. 3.Guidance on the impact evaluation of AI interventions · UK Government