Practical guide
How to create an AI experiment readout
An AI experiment readout should let a decision maker reconstruct what was tested, what happened, what remains uncertain, and why the proposed next step follows. It is not a victory deck. A useful readout preserves failed checks, implementation defects, missing data, and alternative explanations alongside the result.
An AI experiment readout should let a decision maker reconstruct what was tested, what happened, what remains uncertain, and why the proposed next step follows. It is not a victory deck. A useful readout preserves failed checks, implementation defects, missing data, and alternative explanations alongside the result.
Source and claims boundary
The UK Cabinet Office Test, Learn and Grow Playbook treats assumptions as beliefs not yet proved by evidence and recommends combining evidence, exposing uncertainty, and using findings to make the next decision. HM Treasury's TIGER guidance calls for transparent reporting of the intervention, method, comparison groups, outcomes, data collection, missingness, and uncertainty. The NIST AI Risk Management Framework adds AI-specific expectations for documented roles, testing, monitoring, uncertainty, and risk treatment. Together they support a disciplined readout, but they do not prove that an AI intervention caused an observed change or that a test is safe to scale. Cabinet Office: Gathering and using evidence HM Treasury: TIGER reporting guidance NIST: AI RMF Core
Sources: [1] [2] [3]
Build the readout with SIGNAL
### S — State the decision and preregistered claim
Open with the decision, its owner, deadline, and available choices: scale, continue, repair, restrict, or stop. Quote the original hypothesis and success, safety, and stop criteria. State the unit of assignment, eligible population, test period, AI system and version, workflow, comparison, and intended outcome.
Separate three claims that are often collapsed: whether the intervention was delivered, whether the measured outcome differed, and whether the difference can reasonably be attributed to the intervention. A test can succeed operationally without demonstrating an outcome, or show an outcome while failing a safety gate.
### I — Inventory assignment, exposure, and implementation
Report how units entered the experiment, how assignment was produced, and whether it remained intact. Give counts for eligible, assigned, exposed, completed, analyzed, excluded, and missing records by arm. Describe deviations, crossovers, repeat exposure, outages, model changes, prompt changes, manual overrides, and concurrent initiatives.
Implementation measures belong before outcome charts. Without them, a null result may mean the intervention had no effect, was not delivered, or was measured incorrectly. A favorable result may reflect selective exposure rather than the intended workflow.
### G — Give results with denominators and uncertainty
For every primary and guardrail measure, show the definition, source, unit, period, numerator, denominator, estimate by arm, difference, uncertainty interval where appropriate, and missingness. Show absolute values as well as relative movement. Label exploratory analyses and subgroup slices that were not specified in advance.
Do not translate statistical significance into business importance or causal certainty. Report practical magnitude, decision threshold, and uncertainty. If the design does not support causal inference, use descriptive language such as “observed difference” rather than “impact.”
### N — Name counter-evidence and alternative explanations
Include quality failures, complaints, privacy or security events, unsupported AI outputs, downstream rework, burden, and unequal effects. Test credible explanations such as seasonality, learning effects, case mix, novelty, instrumentation changes, attrition, and contamination between arms.
Record what the analysis could not see. Absence of a logged incident is not evidence that no harm occurred. A guardrail with poor coverage should be “not evidenced,” not “passed.”
### A — Account for changes and analytical choices
List every material departure from the plan: changed metric, sample, duration, model, exclusion, transformation, or stopping rule. Preserve the original result and show the sensitivity of the conclusion to reasonable analytical choices. Link calculations to versioned code, extracts, and an analysis log.
AI may help check arithmetic, compare narrative to tables, or enumerate missing fields. It must not invent observations, choose favorable exclusions, or mark its own analysis as independently verified.
### L — Link evidence to a bounded next action
End with a disposition and conditions, not a generic recommendation. State which gate passed, failed, or remained untested; the accountable decision maker; the action; the population and duration; monitoring; rollback trigger; open questions; and next evidence date. Retain dissent and the reason rejected options were not chosen.
Readout artifact
| Field | Required evidence | | --- | --- | | decision | owner, deadline, choices, consequence of delay | | hypothesis | original claim, mechanism, thresholds, stop rules | | design | eligibility, assignment, arms, period, system version | | implementation | assigned, exposed, completed, crossovers, defects | | results | definitions, denominators, estimates, uncertainty, missingness | | risks | incidents, errors, burden, affected groups, counter-evidence | | deviations | plan changes, reason, approver, sensitivity | | disposition | scale, continue, repair, restrict, or stop; conditions | | evidence packet | extracts, code, logs, approvals, version and expiry |
Worked example: synthetic service-summary experiment
A service team tests an AI-assisted summary tool for six weeks. The decision is whether to extend it beyond one queue. The preregistered primary measure is median preparation time per completed case. Guardrails are factual-error rate in a blinded quality sample, missing-attachment rate, sensitive-data incidents, and staff-reported correction burden. The scale rule requires a meaningful timing reduction with no deterioration beyond defined quality limits.
Of 1,020 eligible cases, 510 are assigned to each arm. The readout shows that 64 assisted cases never receive the feature during a two-day outage, 18 control cases are processed by staff who saw assisted examples, and 27 cases lack a final timing event. The primary analysis remains by original assignment; an exposure analysis is labeled exploratory.
Median preparation time is 11.2 minutes in the assisted arm and 13.0 minutes in control, an observed difference of 1.8 minutes. The readout includes the interval estimate and case-mix distribution rather than reporting only a 14% relative change. A blinded review finds 8 factual errors in 120 assisted cases and 3 in 120 control cases. The quality difference is imprecise but breaches the preselected guardrail. Two assisted summaries also omit attachments because the integration did not pass attachment metadata.
The team tests whether case complexity explains the timing result and finds the arms broadly balanced on the preregistered categories. It cannot rule out learning spillover or estimate downstream resolution quality. The disposition is repair and retest, not scale. The workflow remains limited to the current queue; the owner adds source-link and attachment checks, requires correction logging, and schedules a blinded retest. The readout preserves the timing finding, the quality failure, the outage, and the unresolved transfer question.
Product boundary
Two Prune can place a participant in a realistic assessment where they must interpret an experiment packet, verify calculations, surface adverse evidence, and write a bounded readout. It reports evidence about observed assessment performance. It does not run an organization's experiment, validate causality, approve deployment, certify risk controls, predict workplace outcomes, or make employment decisions.
Sources
Cabinet Office: Test, Learn and Grow — gathering and using evidence
- HM Treasury: Transparency in Government Evaluation Research
- NIST: AI Risk Management Framework Core
Sources: [1] [2] [3]
Sources
- 1.Cabinet Office: Gathering and using evidence · UK Government
- 2.HM Treasury: TIGER reporting guidance · UK Government
- 3.NIST: AI RMF Core · National Institute of Standards and Technology