Measuring the impact of an AI programme

State the decision the evaluation must inform

Begin with a real decision: continue a pilot, change the learning design, expand to another workflow, fund a control, or stop an intervention. Define the population, tasks, period, expected mechanism, business-as-usual comparison, material outcomes, and possible harms. A programme containing several different workflows may need separate ladders because the causal paths and evidence differ.

The UK government’s 2026 guidance on evaluating AI interventions recommends early evaluation design, a specified theory of change, a defined baseline, iterative evaluation, and attention to unintended and differential effects. It concerns public-sector interventions, so organizations should adapt its principles rather than treat it as a universal corporate standard.

Do not promise a causal conclusion before choosing a design that can support one. A before-and-after change may reflect workload, seasonality, staffing, policy, model updates, or selection effects. Label descriptive, contributory, and causal findings differently.

Sources: [1]

Construct the impact contribution ladder

Level one records inputs: people, time, tools, learning, support, and controls. Level two records outputs: access provision, completed practice, and workflows configured. Level three records observed behavior in defined tasks, such as source checking, correction, escalation, or revision. Level four records work outcomes, such as accuracy on a bounded artifact, cycle time, review burden, or unresolved exceptions. Level five addresses broader organizational outcomes only when evidence and time horizon support them.

For every step, write the assumed mechanism and a disconfirming observation. If training is expected to improve verification, a suitable test must create a fair opportunity to catch a plausible error. If faster drafting merely shifts more checking to reviewers, speed is not a complete benefit measure.

The OECD AI-WIPS programme studies AI’s effects on work, productivity, and skills and emphasizes responsible, human-centered analysis. Its work supports examining several dimensions, but it does not supply evidence that a particular company programme caused an outcome.

Sources: [2]

Worked artifact: a synthetic proposal-review programme

Imagine a synthetic professional-services firm training twenty proposal specialists to use AI for requirement extraction and red-team review. The funding decision is whether to run a larger controlled cohort. The baseline uses two historical-style synthetic cases completed with the existing workflow.

The ladder records attendance and practice completion at level two, but does not call them impact. Level three measures whether participants identify authoritative requirements, challenge unsupported claims, and preserve unresolved issues. Level four measures material requirement omissions, reviewer correction time, and the usefulness of the final compliance matrix. It also records false alarms and cases where the task provided no opportunity to demonstrate a behavior.

After the programme, the cohort finishes one case faster and catches more seeded contradictions, but reviewers spend longer resolving duplicated warnings. Because the group is small and the cases are synthetic, the team reports a task-bounded directional finding: the training may have changed verification behavior, while net review burden remains uncertain. It does not claim improved win rates, workforce productivity, or future performance.

The next decision is to revise warning triage and run a larger comparison with a contemporaneous business-as-usual group. The artifact retains case versions, scoring anchors, reviewer disagreements, timing rules, missing observations, and deviations. The example is synthetic and is not evidence of Two Prune customer results.

Protect the evaluation from convenient metrics

Pre-register the main questions and decision rules where practical. Keep adoption, capability, quality, efficiency, risk, and participant experience separate. Report denominators and missingness. Segment only when sample size and design permit; do not produce precise subgroup claims from sparse observations.

Track model, prompt, interface, policy, and workflow changes during the evaluation. If the intervention changes substantially, preserve versioned phases rather than pooling incompatible evidence. Include negative and null findings, and distinguish data unavailable from a failed outcome.

Verify every rung

Trace each metric to a source, calculation, and owner. Reperform material calculations and review outliers. Confirm that assessment cases created the intended evidence opportunities and that reviewers applied anchors consistently. Compare the written theory with observed workarounds and downstream review.

This ladder makes claim strength visible and ties measurement to a decision. It does not establish mature benchmarks, prove return on investment, or establish causality without an appropriate design. Two Prune can contribute task-bounded process-and-output evidence for human interpretation; it does not determine programme funding or employment outcomes.