How to evaluate a programme with AI

Reconstruct the theory of change

Start with the programme’s problem, inputs, activities, outputs, expected outcomes, causal mechanisms, assumptions, and contextual conditions. Compare the intended model with what was actually implemented. A result cannot be interpreted well if delivery varied or the mechanism was never activated.

The 2026 UK Magenta Book defines evaluation as systematic assessment of intervention design, implementation, and outcomes. It emphasizes theory of change, user needs, context, and selecting methods that fit the evaluation question. Its quality guidance distinguishes observing an outcome from attributing that outcome to an intervention. These principles apply here as method references, not evidence about a specific programme.

Use an evaluation map:

| Question | Evidence | Design contribution | Limitation | |---|---|---|---| | Was the programme delivered as planned? | Attendance, practice records, facilitator logs | Process assessment | Records may omit informal support | | Did target behavior change? | Comparable work samples before and after | Outcome change | Other events may influence change | | Did the programme cause the change? | Credible comparison or counterfactual | Attribution | Depends on comparability and design integrity | | For whom and why did it vary? | Subgroup, context, and qualitative evidence | Mechanism and context | Small groups limit interpretation |

Test implementation before outcomes

Describe reach, dosage, fidelity, adaptations, access barriers, and participant experience. Distinguish planned adaptation from delivery failure. If the practice component was removed in one location, pooled outcome results may hide a meaningful implementation difference.

This synthetic example concerns a six-week AI-verification programme. Eighty people enrolled, but the evaluation does not treat enrollment as capability. Work-sample completion was 62 before and 51 after; 44 records can be matched under the approved protocol. Two teams received manager coaching while one did not. Those facts constrain which comparisons are possible.

Ask AI to reconcile identifiers, surface missingness patterns, and compare implementation logs. Review transformations and protect participant data. Do not infer that non-completers performed poorly.

Match outcome claims to the design

Define outcomes before analysis where possible and preserve changes to the plan. Check measurement timing, consistency, missing data, and whether instruments offer the same opportunity to demonstrate the behavior.

A weak evaluation says, “Scores improved after training, so the programme worked.” A before-and-after change alone does not rule out practice effects, selection, manager support, changes in task difficulty, or concurrent initiatives. A stronger statement says, “Matched work samples improved on the defined verification criterion; because no credible comparison was available and coaching differed by team, the evaluation cannot isolate the programme’s contribution.”

Where a comparison exists, assess whether groups were formed credibly, exposed to similar external conditions, and measured consistently. Report deviations and sensitivity checks. If the design cannot answer attribution, focus on implementation, observed change, mechanisms, and evidence needed next.

Integrate context and unintended effects

Examine who participated, who did not, where delivery varied, and what external changes occurred. Qualitative evidence can explain mechanisms and implementation, but do not use a few quotations as prevalence estimates. Record positive and negative unintended effects separately.

For the synthetic programme, manager coaching may explain team differences or may be a marker for other team conditions. Treat it as a hypothesis unless the design supports a stronger conclusion. A follow-up design could predefine coaching exposure and collect comparable evidence.

Verify the evaluation report

Before release, confirm:

The artifact is the evaluation report plus theory-of-change map, implementation record, analysis plan, evidence table, limitation register, and reproducible calculations. Preserve code and decisions so a later reviewer can reconstruct the result.

• the theory of change and assumptions are explicit;
• intended and actual implementation are compared;
• outcome definitions and analysis changes are recorded;
• missing data and participation patterns are visible;
• comparison quality and counterfactual limits are stated;
• causal wording does not exceed design strength;
• context and unintended effects are considered;
• each finding maps to evidence; and
• recommendations distinguish what is known from what should be tested.

Sources and claim map

The programme, numbers, teams, and findings are synthetic. This resource does not establish programme impact, individual development, or broad transfer.

• HM Treasury and Evaluation Task Force, Magenta Book 2026: supports theory-of-change, implementation, outcome, context, and design-to-question guidance.
• UK Government, Quality in policy impact evaluation: supports distinguishing outcome change from causal attribution and assessing counterfactual quality.

Sources: [1] [2]