How to evaluate a product experiment with AI

Reconstruct the experimental contract

Record the user problem, hypothesis, target population, unit of assignment, intervention, comparison, primary outcome, guardrails, minimum observation window, stopping rule, and planned segments. State the decision the result will inform.

HM Treasury’s 2026 Magenta Book describes evaluation as systematic assessment and explains counterfactual comparison. Its Test and Learn guidance recommends identifying uncertain assumptions and preserving credible evaluation designs. Although written for government interventions, the causal discipline transfers to product experiments without validating a commercial feature.

Reconcile eligibility, assignment, and exposure

Create a participant flow:

eligible → assigned → feature available → actually exposed → outcome observed → analyzed

Assignment and exposure answer different questions. Comparing only users who chose to engage can introduce selection after assignment. Report exclusions and missing outcomes by arm.

Worked synthetic experiment

Suppose a team tests an inline evidence panel in an assessment-review workspace. The hypothesis is that showing relevant evidence beside a report reduces time spent locating source material without increasing correction errors.

| Measure | Control | Treatment | | --- | ---: | ---: | | Assigned reviewers | 120 | 120 | | Opened eligible report | 102 | 99 | | Actually saw panel | 0 | 91 | | Median review time | 31 min | 27 min | | Reports requiring later correction | 4 | 7 |

All figures are synthetic. The time difference is an observed comparison, not yet a causal or launch conclusion. Exposure was incomplete, sample sizes are modest, and the correction guardrail moved in an unfavorable direction.

Check design fidelity

Verify randomization or allocation code, sample-ratio balance, feature delivery, event definitions, timestamps, repeated-user handling, and contamination between arms. Check whether a deployment, training change, or report mix differed during the window.

Ask AI to compare the analysis dataset with assignment logs and produce a discrepancy list. It must not silently discard cases that make the result less favorable.

If assigned group counts differ unexpectedly, pause interpretation and inspect eligibility, randomization, logging, and joins before treating the outcome analysis as trustworthy.

Analyze the planned outcome and guardrails

Use the predeclared denominator and method. Show absolute values, differences, uncertainty, and data quality. Treat exploratory segments as hypotheses for future testing. Review guardrails even if the primary metric improves.

For the example, a responsible summary is: “Treatment reviewers completed observed reviews faster, while later corrections were numerically higher. The dataset does not yet support rollout. Investigate correction types, panel exposure failures, and report-mix balance, then decide whether to extend or redesign the test.”

Challenge alternative explanations

List plausible causes of the pattern: easier reports in one arm, experienced reviewers unevenly distributed, incomplete panel loading, altered reviewer behavior, or random variation. Define one discriminating check for each. Separate mechanism evidence from outcome movement.

Make the decision reversible

Choose continue, modify, stop, or scale only within the evidence boundary. Record what evidence would reverse the decision and which owner approves it. Preserve the assignment file, queries, analysis version, and deviations.

Verification checklist

Confirm the hypothesis and primary outcome predate analysis; reconcile flow counts; verify allocation and instrumentation; inspect missingness; report guardrails; avoid favorable-only segmentation; test alternatives; preserve reproducibility; and keep claims within the observed context.

Sources and claim map

The feature, data, and result are synthetic. This resource does not claim a causal effect, product improvement, validated benchmark, or outcome outside the experiment.

• HM Treasury, Magenta Book 2026: supports systematic evaluation, counterfactual reasoning, method fit, and transparent interpretation.
• HM Treasury, Test and Learn: supports testing uncertain assumptions, structured iteration, and separating exploration from stronger impact evaluation.

Sources: [1] [2]