Practical guide

AI data analysis notebook verification checklist

A data analysis notebook is verified when a reviewer can identify the exact inputs, restart the environment, execute the accepted path in order, reproduce material values, challenge decision-facing claims, and see who owns unresolved limitations. Visible code and polished charts are not enough if hidden state, stale data, or an ambiguous denominator can change the conclusion.

By Two Prune

A data analysis notebook is verified when a reviewer can identify the exact inputs, restart the environment, execute the accepted path in order, reproduce material values, challenge decision-facing claims, and see who owns unresolved limitations. Visible code and polished charts are not enough if hidden state, stale data, or an ambiguous denominator can change the conclusion.

Freeze the verification packet

Record these fields before testing:

If a source, environment, or decision changes during review, create a new packet. Do not combine a current output with evidence from an earlier run.

Jupyter’s notebook format records ordered cells, source, outputs, and metadata, but external files, dependencies, execution state, and services still require control. The UK Government Analysis Function’s Reproducible Analytical Pipelines guidance emphasizes version control, minimized manual steps, peer review, documentation, input validation, testing, logging, and dependency management. RESTART applies those principles to a single notebook review; passing it does not establish that the underlying analysis is suitable for every decision. Jupyter nbformat: The notebook file format UK Government Analysis Function: Reproducible Analytical Pipelines

  • [ ] notebook identifier, version, author, reviewer, and decision supported
  • [ ] accepted source files or queries, extraction times, owners, and access rules
  • [ ] runtime, package lock, configuration, timezone, locale, and random seeds
  • [ ] release cells and outputs, separated from exploratory history
  • [ ] material claims, acceptance tests, and review deadline
  • [ ] approved location for restricted data, outputs, and review evidence

Sources: [1] [2]

Run the RESTART checklist

### R — References and raw inputs

### E — Environment and execution order

### S — Structure and transformations

### T — Tests and calculations

### A — AI contribution and analyst authority

### R — Results and claim traceability

### T — Transfer, approval, and residual uncertainty

  • [ ] Every source has a stable identifier, version or extraction time, owner, population, and known limitation.
  • [ ] Raw inputs are read-only or otherwise protected from accidental notebook writes.
  • [ ] Sensitive data are handled only in approved tools and locations.
  • [ ] The notebook reports data-as-of time and expected source latency.
  • [ ] Representative records can be traced from source through the final output.
  • [ ] Runtime and dependency versions are machine-readable where possible.
  • [ ] Credentials and secrets are not embedded in cells or outputs.
  • [ ] A restart-and-run-all execution succeeds without manual variable creation.
  • [ ] Cell order matches dependency order; out-of-order execution cannot silently change the result.
  • [ ] Randomness, locale, timezone, and external service versions are controlled or disclosed.
  • [ ] Each join states its keys, expected cardinality, unmatched treatment, and duplication test.
  • [ ] Filters, exclusions, imputation, parsing, deduplication, and unit conversions are explained.
  • [ ] Material steps report input and output row counts.
  • [ ] Repeated logic is placed in tested functions rather than copied across cells.
  • [ ] Manual edits or external spreadsheet steps are removed or documented with evidence.
  • [ ] Numerators, denominators, cohorts, grains, windows, and units are explicit.
  • [ ] At least one headline value is reproduced independently.
  • [ ] Boundary cases, nulls, unexpected categories, empty inputs, and duplicate keys are tested.
  • [ ] A small fixture has a known expected result.
  • [ ] Failed assertions stop or clearly invalidate dependent outputs.
  • [ ] Material AI-generated code or interpretation is distinguishable from source facts and analyst decisions.
  • [ ] Accepted suggestions record the verification performed; rejected suggestions remain explainable when material.
  • [ ] AI output did not introduce an unapproved data transfer, package, source, or claim.
  • [ ] An accountable analyst owns the method, interpretation, and release.
  • [ ] Plausible explanations are not treated as evidence that code executed correctly.
  • [ ] Each decision-facing statement links to a source, calculation, and cell.
  • [ ] Descriptive, predictive, and causal claims are labelled distinctly.
  • [ ] Charts expose units, populations, time periods, missingness, and material uncertainty.
  • [ ] Alternative definitions or sensitivity tests are shown where they could change the decision.
  • [ ] Rejected or unresolved claims cannot reappear in the summary through copied text.
  • [ ] The final handoff answers the original question and states what it cannot answer.
  • [ ] Reviewer findings have owners, dispositions, due dates, and closure evidence.
  • [ ] The accepted notebook, rendered output, environment, and source manifest share a versioned release reference.
  • [ ] Exported tables and charts match the verified cells.
  • [ ] Residual uncertainty is visible to the decision maker and has a next evidence gate.

Copyable verification record

| Check ID | Notebook element | Source or expected rule | Test performed | Expected | Observed | Disposition | Owner / approver | Residual uncertainty | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | NB-01 | input, cell, transformation, claim, or export | exact version and rule | trace, rerun, fixture, recalculate, or challenge | measurable result | retained evidence | pass / repair / hold / remove | named people | remaining question |

Attach the source manifest, environment definition, execution log, fixture data, assertion results, independent calculation, claim register, review notes, and final approval. A screenshot or HTML export can preserve presentation, but it cannot replace the executable notebook and its controlled inputs.

Worked test: quarterly renewal notebook

A synthetic notebook compares renewal rates for small and large customers. It contains clean charts and runs when opened in the author’s existing session.

The environment test fails after restart because an earlier uncommitted cell created eligiblecontracts. The author moves the transformation into the accepted path and pins the package versions. The reference test then finds that the contract extract lacks an extraction timestamp; a dated snapshot and owner are added.

The structure test detects an account-level join to contract-level renewals. One account with three contracts creates three copies of its segment attributes and changes the numerator. A cardinality assertion fails, the join moves to contract ID, and unmatched records receive an explicit category rather than being dropped.

The calculation test finds that the numerator counts contracts renewed in the quarter while the denominator counts contracts expiring in the quarter. The notebook repairs the cohort definition and includes a ten-row fixture with two renewed, one not renewed, and one still inside the late-reporting window. The expected rate is stated before the code runs.

An AI assistant suggests filling unknown segment values from company size. The analyst rejects the suggestion because the size field is stale and could move contracts between comparison groups. Unknown remains visible. A sensitivity cell shows that excluding immature contracts narrows the observed gap from 6.2 to 2.1 percentage points.

The claim test removes “onboarding causes small-business churn.” The verified handoff states only that recorded renewal differs in the mature cohort and that available evidence does not identify why. The completed packet contains the repaired join, fixture, restart log, sensitivity result, rejected imputation, qualified claim, reviewer approval, and next investigation.

Independent release challenge

Give a reviewer who did not author the notebook one headline claim. Ask them to locate the source version, restart and execute the path, reproduce the value independently, explain a material assumption, and predict the result of an edge case. Expand review when they need undocumented guidance or find any source, grain, calculation, or export mismatch.

Hold release when inputs cannot be frozen, hidden state changes the result, the accepted path does not run cleanly, a material join or denominator lacks a test, sensitive data leave approved controls, a claim cannot be traced, or exported outputs differ from verified cells.

Two Prune can place notebook verification inside a realistic AI-enabled work assessment and evaluate how a participant navigates data, uses AI, tests calculations, manages uncertainty, and communicates a deliverable. It does not operate analytical pipelines or make employment decisions.

Sources

Jupyter nbformat: The notebook file format

  • UK Government Analysis Function: Reproducible Analytical Pipelines

Sources: [1] [2]

Sources

  1. 1.Jupyter nbformat: The notebook file format · nbformat.readthedocs.io
  2. 2.UK Government Analysis Function: Reproducible Analytical Pipelines · analysisfunction.civilservice.gov.uk