Practical guide
What evidence is needed to measure AI adoption?
Evidence for AI adoption should show who had a real opportunity to use an approved tool, what happened in a named workflow, how the records were produced, and what the evidence cannot establish. A usage total alone is not adoption evidence, and adoption is not the same as capability, work quality, or business impact. Use PACK: pin the measurement question, assemble an evidence contract, check.
Evidence for AI adoption should show who had a real opportunity to use an approved tool, what happened in a named workflow, how the records were produced, and what the evidence cannot establish. A usage total alone is not adoption evidence, and adoption is not the same as capability, work quality, or business impact. Use PACK: pin the measurement question, assemble an evidence contract, check coverage and integrity, and keep interpretation bounded. The deliverable is an auditable evidence packet with denominators, provenance, missingness, and context.
P: Pin one measurement question
Begin with the decision that the evidence must inform. “How many accounts exist?”, “Are eligible people returning to an approved tool?”, “Is use occurring in the target workflow?”, “Can people handle a realistic AI-assisted task?”, and “Did an AI intervention cause an outcome?” are different questions.
Define the construct in observable terms. For repeated use, specify the useful action, eligible population, observation window, approved tools, and treatment of service accounts, tests, and automated events. For workflow integration, specify the task and opportunity. For capability, use task-bounded work evidence. For impact, use an evaluation design suited to causal questions.
The UK government's 2026 guidance on impact evaluation of AI interventions says evaluation should be designed early, tailored to context, and proportionate to risk and learning needs. It distinguishes assessing outcomes and attribution from simple activity reporting. Use it to frame an evaluation decision, not to relabel usage telemetry as impact evidence.
A: Assemble an evidence contract before collecting data
Create an evidence contract for every measure:
| Field | Question | |---|---| | Construct | What exactly is being measured? | | Decision | What action could this evidence change? | | Population | Who is eligible, included, excluded, and why? | | Opportunity | Who could reasonably perform the measured action? | | Numerator | Which events or records qualify? | | Denominator | Which population makes the rate interpretable? | | Window | When does measurement start and end? | | Source | Which system or collection method creates the record? | | Provenance | Which version, query, configuration, and extraction produced it? | | Quality rules | What counts as complete, valid, unique, consistent, timely, and accurate? | | Missingness | What is absent, why might it be absent, and how is it handled? | | Context | Which releases, policies, training, staffing, or outages affect interpretation? | | Privacy | What minimisation, access, retention, and aggregation rules apply? | | Limit | Which claims cannot be made from this measure? |
Specify these fields before looking at trends. Otherwise teams may redefine an active user, denominator, or time window after seeing an inconvenient result.
Evidence sources and what they can support
Use multiple sources only when each has a clear role. Common classes include:
Do not join records merely because identifiers exist. Document why linkage is necessary, which records fail to link, who can access the result, and whether linkage changes the privacy or employment context.
- eligibility and access records: approved population, licence status, provisioning, and technical access;
- tool events: defined actions, sessions, errors, feature use, and return activity;
- workflow records: opportunities, task type, handoffs, approvals, and completed work;
- reviewed artifacts: source traces, corrections, defects, escalation, and work quality under a stated sampling method;
- realistic simulations: observable framing, evidence use, verification, judgment, and communication under known conditions;
- surveys and interviews: experience, barriers, confidence, and explanations, subject to response and selection effects;
- operational outcomes: time, rework, service, cost, or risk indicators with contextual and evaluation controls.
C: Check coverage, integrity, and meaning
The Government Data Quality Framework treats quality as fitness for purpose and asks users to consider completeness, uniqueness, consistency, timeliness, validity, and accuracy. Apply those dimensions to adoption evidence.
Coverage checks should reconcile licences, eligible people, active accounts, workflow opportunities, and observed events. Integrity checks should remove test and service activity, identify duplicates, validate timestamps, preserve query versions, and detect instrumentation changes. Meaning checks should confirm that an event represents the intended action rather than a page load or automatic process.
Document missingness instead of silently excluding it. A missing event may mean the action did not happen, logging failed, the person used an unobserved channel, the task never arose, consent or privacy rules limited collection, or extraction failed. These explanations lead to different decisions.
NIST's AI RMF Measure playbook calls for context-relevant metrics, documented limitations, suitable testing, monitoring, human expertise, and attention to data quality and representativeness. The playbook is aimed at AI risk measurement, but its measurement discipline helps expose weak adoption evidence.
K: Keep interpretation bounded and decision-ready
Report evidence as a set of linked findings, not one maturity score. Show counts and denominators together. Annotate tool releases, policy changes, training dates, outages, staffing changes, and extraction revisions. Separate direct observation from inference.
Use an interpretation ledger:
| Finding | Supported interpretation | Alternative explanation | Missing evidence | Decision | |---|---|---|---|---| | 72 of 120 eligible people completed a defined action | Recorded activation was 60% in the window | Some eligible people had no relevant task | Opportunity records | Investigate opportunity and access | | 31 people used the tool in a named workflow | At least 31 linked records show workflow use | Linkage misses off-platform work | Linkage coverage audit | Repair instrumentation | | 8 of 15 sampled artifacts needed source correction | Corrections occurred in the reviewed sample | Risk-based sample overrepresents difficult work | Sampling frame | Do not generalise prevalence |
Do not call a change causal unless the design supports attribution. Do not infer capability from frequency or quality from duration. Do not infer non-use from a missing log without an opportunity and logging check. Report when evidence is too weak for action.
Worked evidence packet
A synthetic organisation has 120 employees eligible for an approved assistant. The licence table shows 118 provisioned accounts; two are pending accessibility configuration. Event logs contain 79 accounts, but seven belong to testers and automated quality checks. After applying the pre-registered active-use rule, 72 eligible people completed a defined useful action during four weeks.
The target workflow system records 64 proposal-review opportunities. Thirty-one records link to approved-tool activity, 18 show no linked event, and 15 cannot be joined because an identifier changed during a system migration. A reviewed sample of 15 artifacts includes eight selected for known complexity, so its correction rate cannot estimate the rate for all proposals. Three participants complete a separate synthetic reconciliation exercise, which provides task evidence but not a population estimate.
The packet does not report “60% adoption” as the whole conclusion. It reports activation among the eligible population, workflow-linkage coverage, the migration-related missingness, the non-representative artifact sample, and the distinct simulation evidence. The decision is to repair identifiers, define workflow opportunity consistently, and repeat the evidence extraction before changing programme investment.
Evidence packet quality check
Before release:
- Reconcile the population and opportunity denominators to named source systems.
- Reproduce the extraction from a versioned query or procedure.
- Inspect a sample of qualifying and excluded records.
- Verify dates, time zones, duplicates, service accounts, and automated events.
- Record logging changes and unobserved channels.
- Test whether the joined records preserve the intended meaning.
- State sampling and missingness rules next to the finding.
- Separate adoption, capability, quality, and impact conclusions.
- Require a human owner to approve the interpretation and action.
Two Prune boundary
Two Prune can provide task-bounded assessment evidence from a realistic simulation and explain observed process and deliverable evidence. Its assessment evidence can complement adoption telemetry when the decision concerns applied AI fluency. It does not measure organisation-wide adoption by itself, establish causal impact, prove learning transfer, certify a workforce, predict future workplace performance, or make employment decisions.
Primary sources
UK Government, Guidance on the impact evaluation of AI interventions
- NIST AI RMF Playbook, Measure
- UK Government, Government Data Quality Framework guidance
Sources: [1] [2] [3]
Sources
- 1.UK Government, Guidance on the impact evaluation of AI interventions · UK Government
- 2.NIST AI RMF Playbook, Measure · National Institute of Standards and Technology
- 3.UK Government, Government Data Quality Framework guidance · UK Government