Practical guide
AI experiment readout verification checklist
Use this checklist after an AI experiment readout has been drafted and before it informs a scale, continue, repair, restrict, pause, or stop decision. It verifies whether a reader can reconstruct the test, reproduce the analysis, see adverse evidence, and understand why the proposed action follows. A clear presentation does not compensate for a broken assignment record, an untraceable.
Use this checklist after an AI experiment readout has been drafted and before it informs a scale, continue, repair, restrict, pause, or stop decision. It verifies whether a reader can reconstruct the test, reproduce the analysis, see adverse evidence, and understand why the proposed action follows. A clear presentation does not compensate for a broken assignment record, an untraceable denominator, or an unreported guardrail failure.
What the checklist can verify
The UK Cabinet Office Test, Learn and Grow Playbook says testing should be tied to decisions, planned around assumptions and hypotheses, record results, consider risk and ethics, and use learning to decide whether to iterate, scale, pause, or stop. HM Treasury's TIGER guidance calls for transparent descriptions of methods, comparisons, outcomes, data collection, missingness, and uncertainty. The NIST AI Risk Management Framework calls for documented roles, monitoring, testing, uncertainty, and risk treatment for AI systems. These sources support verification discipline. They do not establish that a test caused an observed change or that an AI workflow is safe to expand. Cabinet Office: Testing, learning and iterating HM Treasury: TIGER reporting guidance NIST: AI RMF Core
Sources: [1] [2] [3]
How to use the VERIFY checklist
Mark each item pass, fail, not applicable, or not evidenced. A pass requires a linked artifact or a reproducible check. Assign every failure an owner, due date, and consequence for the decision. Do not average a critical failure into an overall score.
### V - Version, decision, and test boundary
Stop condition: hold the decision when the test version or decision boundary cannot be identified, or when the readout substitutes a different hypothesis after results were known without showing the change.
### E - Eligibility, assignment, and exposure
Stop condition: do not issue an arm comparison when assignment integrity is unknown or the reported denominator cannot be reconciled.
### R - Results and recalculation
Stop condition: hold a claim that depends on a calculation that cannot be reproduced or on a measure whose definition changed without a visible bridge.
### I - Integrity, risks, and adverse evidence
Stop condition: return the readout when a material adverse result is omitted, downgraded without authority, or buried outside the decision view.
### F - Fidelity to the plan and alternative explanations
Stop condition: reject a causal or general claim when the design can support only an observed difference in the tested setting.
### Y - Yield a bounded decision and evidence trail
Stop condition: do not treat the readout as approval when the accountable authority, decision conditions, or rollback route is missing.
- [ ] The readout names the decision, accountable decision maker, deadline, and credible alternatives.
- [ ] The original hypothesis, mechanism, primary measure, guardrails, success threshold, and stop rule are preserved.
- [ ] The eligible population, assignment unit, arms, period, workflow, AI system, model or tool version, prompts or configuration, and comparison are identified.
- [ ] The readout distinguishes implementation, observed outcome, attribution, and transfer claims.
- [ ] Every source extract, analysis file, chart, and issued readout has a version and retained evidence reference.
- [ ] Eligible, excluded, assigned, exposed, completed, analyzed, missing, and withdrawn units reconcile by arm.
- [ ] Assignment generation and any stratification or blocking can be reproduced.
- [ ] Crossovers, failed exposure, repeat exposure, contamination, outages, and manual overrides are visible.
- [ ] Exclusions and missingness rules match the plan and are applied consistently.
- [ ] Technical failures and participant burden remain in implementation reporting even if excluded from an outcome calculation.
- [ ] Each measure has a definition, source, unit, period, numerator, denominator, and extraction date.
- [ ] A reviewer can reproduce all material totals, rates, differences, intervals, and transformations.
- [ ] Absolute values appear alongside relative changes.
- [ ] Primary, secondary, guardrail, subgroup, and exploratory analyses are clearly labeled.
- [ ] Charts agree with tables and narrative, including axes, filters, rounding, and uncertainty.
- [ ] Practical magnitude is compared with the decision threshold rather than statistical significance alone.
- [ ] Quality failures, unsupported AI outputs, privacy or security events, complaints, accessibility issues, rework, burden, and unequal effects are reported under the same materiality rule as benefits.
- [ ] Guardrail coverage and monitoring gaps are quantified.
- [ ] Events caught by a control are not described as events that never occurred.
- [ ] Absence of a recorded incident is not presented as proof of safety.
- [ ] Affected people, control owners, containment, residual uncertainty, and escalation routes are named.
- [ ] Changes to metrics, sample, duration, model, prompts, analysis, exclusions, or stopping rules are listed with timing, reason, and approval.
- [ ] The original analysis remains available and sensitivity checks show whether reasonable choices change the conclusion.
- [ ] Credible alternative explanations such as case mix, seasonality, novelty, learning, attrition, instrumentation change, or concurrent initiatives are tested or marked unresolved.
- [ ] Causal wording matches the design; descriptive designs use descriptive claims.
- [ ] Limits on population, setting, duration, and workplace transfer are stated.
- [ ] The disposition is scale, continue, repair, restrict, pause, or stop, with a clear rationale.
- [ ] The chosen action specifies owner, population, duration, conditions, monitoring, rollback trigger, and next review date.
- [ ] Rejected options, dissent, unresolved questions, and evidence still needed are retained.
- [ ] The evidence packet contains the plan, assignment record, extracts, analysis, deviations, incidents, approvals, and issued version.
- [ ] Evidence expiry is triggered by material changes to the workflow, AI system, data, population, control, or objective.
Verification record
| Area | Result | Evidence IDs | Failure or uncertainty | Owner | Due date | | --- | --- | --- | --- | --- | --- | | version and boundary | | | | | | | assignment and exposure | | | | | | | results and recalculation | | | | | | | risks and adverse evidence | | | | | | | fidelity and alternatives | | | | | | | decision and evidence trail | | | | | |
Overall disposition: verified for the stated decision, verified with visible limitations, return for repair, or hold the dependent decision. The disposition applies only to the reviewed version and evidence packet.
Worked verification: synthetic claims-summary test
A product team tests an AI assistant that drafts claim summaries from approved research. Its readout recommends expansion after reporting an 18 percent reduction in drafting time and no material quality change. A reviewer applies VERIFY.
The boundary check passes: the decision, hypothesis, system version, six-week period, and scale threshold are present. The assignment check does not. Of 600 eligible tasks, 300 were assigned to each arm, but an outage prevented 42 assisted tasks from receiving the feature. The headline analysis includes only exposed tasks in the assisted arm while retaining all assigned controls. The reviewer restores the preregistered assignment analysis and labels exposure analysis exploratory.
Recalculation reproduces the timing values but finds that the chart reports a relative difference without the absolute 1.6-minute change. Quality sampling contains 100 tasks per arm. Seven assisted summaries and three control summaries contain unsupported claims. The difference is imprecise but exceeds the preset guardrail. Two assisted summaries also omitted attachments because that input was not logged.
The original deck says the experiment proved productivity improved without quality loss. The verified version instead reports a descriptive timing difference, the failed quality guardrail, outage-related exposure failure, and incomplete attachment evidence. The result is return for repair. The workflow remains bounded while the owner adds source-link and attachment checks, repairs exposure logging, and repeats the blinded quality sample. The checklist does not approve expansion or predict later workplace outcomes.
Two Prune boundary
Two Prune can assess whether a participant verifies a supplied experiment readout, finds evidence gaps, recalculates results, and writes a bounded disposition. It reports observed performance in that assessment. It does not validate an organization's experiment, certify safety or controls, approve deployment, predict workplace outcomes, or make employment decisions.
Sources
Cabinet Office: Test, Learn and Grow - testing, learning and iterating
- HM Treasury: Transparency in Government Evaluation Research
- NIST: AI Risk Management Framework Core
Sources: [1] [2] [3]
Sources
- 1.Cabinet Office: Testing, learning and iterating · UK Government
- 2.HM Treasury: TIGER reporting guidance · UK Government
- 3.NIST: AI RMF Core · National Institute of Standards and Technology