How to evaluate a marketing experiment with AI

Reconstruct the experiment contract

Before looking at the result, record the target population, intervention, comparison, primary outcome, guardrails, unit of assignment, intended sample, start and stop rules, and analysis plan. The 2026 HM Treasury Magenta Book uses the PICOT frame—population, intervention, control, outcome, and time period—as one planning aid and explains the role of counterfactual comparison in impact evaluation. Those principles are broadly useful, though the guidance does not validate a particular marketing test.

Audit assignment and exposure separately

Assignment answers who was allocated to each arm. Exposure answers who actually received or saw the treatment. Do not silently analyze only exposed people if assignment was randomized; that can reintroduce selection bias. Reconcile counts from eligibility through assignment, delivery, exposure, outcome availability, and exclusions.

For a synthetic email test, suppose 20,000 eligible accounts were randomly assigned equally to message A and B. Delivery logs show 9,500 delivered in A and 9,900 in B because one provider rejected more A messages. The experiment readout should show both intention-to-treat results by assignment and a clearly labeled delivery analysis. Comparing only openers would select people based on post-assignment behavior.

Calculate outcomes with uncertainty

Use the predeclared primary metric and denominator. Show absolute rates, absolute difference, relative difference where useful, sample sizes, and an appropriate uncertainty interval. Check implementation and data quality before interpreting statistical output.

Synthetic result:

| Measure | A | B | Difference | | --- | ---: | ---: | ---: | | Assigned | 10,000 | 10,000 | — | | Delivered | 9,500 | 9,900 | -400 | | Primary conversions by assignment | 310 | 330 | +0.20 percentage points for B | | Complaint events | 18 | 39 | +21 for B |

This table alone does not establish that B is preferable. Delivery imbalance, uncertainty, and the complaint guardrail matter.

Check four failure modes

Ask AI to list conclusions that remain valid under alternative denominators and plausible missing-data assumptions. Retain null and adverse findings.

• Peeking: Was the test stopped because a favorable interim result appeared rather than under a stated rule?
• Selection: Were arms comparable at assignment, and were exclusions applied symmetrically?
• Instrumentation: Did tracking or attribution differ by message, channel, browser, or time?
• Multiplicity: Were many outcomes or segments searched and only a favorable one reported?

Write a decision-oriented readout

Use five blocks: design fidelity, primary result, guardrails, limitations, and next decision. A responsible conclusion for the example might be: “B shows a small observed increase in conversion by assignment, while complaint events are higher and delivery differs. Do not declare a winner until uncertainty and delivery imbalance are reviewed; consider a follow-up test with corrected delivery.”

Verification checklist

Match analysis to the preregistered or approved plan; reconcile assignment and exposure; inspect missing outcomes; validate event definitions; report all declared guardrails; distinguish exploratory segments; test alternative explanations; and obtain accountable review before rollout.

Sources and claim map

HM Treasury, Magenta Book: Central Government guidance on evaluation: supports systematic evaluation, early design, counterfactual reasoning, and fit-for-purpose methods.

• HM Treasury, Test and Learn: supports proportionate, iterative, evidence-informed testing and explicit learning designs.

Sources: [1] [2]

Preserve a reproducible record

Retain the approved design, query versions, assignment file, exclusion log, and dated readout. Another analyst should be able to recreate the denominators and understand every departure from plan. Reproducibility does not remove bias, but it makes review and correction possible.

All experiment values and outcomes are synthetic. This resource does not claim a causal effect, assured lift, mature benchmark, or validated marketing result.

Keep the underlying analysis reproducible for independent review.