Practical guide
How to create a data analysis notebook with AI
A useful data analysis notebook is an executable evidence trail, not a transcript of exploratory prompts. It should let a reviewer identify the frozen inputs, understand every transformation, rerun the analysis in a known environment, test the important claims, and separate source facts, AI suggestions, analyst judgments, and final decisions.
A useful data analysis notebook is an executable evidence trail, not a transcript of exploratory prompts. It should let a reviewer identify the frozen inputs, understand every transformation, rerun the analysis in a known environment, test the important claims, and separate source facts, AI suggestions, analyst judgments, and final decisions.
Write the notebook contract first
Before opening an AI assistant or writing analysis code, state:
The contract prevents the notebook from quietly changing purpose as interesting patterns appear. When the question changes, record the change and decide whether it needs a separate analysis.
The Jupyter notebook format stores ordered cells, their source, outputs, and metadata in a structured document. Its specification supports reproducible handoff only when the analyst also controls dependencies, external data, execution order, and hidden state. The UK Government Analysis Function’s Reproducible Analytical Pipelines guidance adds version control, peer review, documentation, input validation, testing, logging, dependency management, and minimized manual steps. These sources inform the notebook method below; neither makes an individual analysis correct. Jupyter nbformat: The notebook file format UK Government Analysis Function: Reproducible Analytical Pipelines
- the decision or question the notebook supports;
- the intended reader and what they may do with the result;
- the source datasets, versions, owners, access conditions, and data-as-of time;
- the unit of analysis, population, important exclusions, and expected grain;
- the output required: exploration, estimate, chart, model, recommendation, or reproducible extract;
- the material risks, including privacy, missingness, leakage, small groups, and unsupported causal interpretation;
- the reviewer, acceptance tests, deadline, and release boundary.
Sources: [1] [2]
Build a six-part evidence spine
Organize the notebook so another analyst can follow the decision path without reading every experimental cell.
### 1. Input manifest
Create a table with one row per source: exact location or identifier, version or extraction timestamp, owner, population, fields used, access rule, checksum where appropriate, and known limitations. Never paste sensitive source data into an unapproved AI tool merely to simplify the workflow.
### 2. Environment capsule
Record the language and runtime, package versions, locale, timezone, random seeds, configuration values, and any service or model version needed to rerun the work. Keep credentials outside the notebook. Prefer a lockfile or environment definition to a prose list when the toolchain supports it.
### 3. Transformation ledger
Use small named steps. For each material transformation, state the input grain, output grain, rule, reason, and verification. Include joins, deduplication, parsing, exclusions, imputation, currency or timezone conversion, derived fields, aggregation, and weighting. Show record counts before and after high-impact steps.
### 4. Analysis and challenge cells
Place each decision-relevant result beside its calculation and challenge. A challenge may be a hand calculation, alternative definition, sensitivity test, holdout, negative control, boundary-value test, or comparison with an independently produced result. Exploration can remain in the notebook, but label it so it cannot be mistaken for a pre-specified test.
### 5. Claim register
For every headline statement, record:
| Claim ID | Statement | Supporting cell and source | Assumptions | Challenge performed | Status | Owner | | --- | --- | --- | --- | --- | --- | --- | | C-01 | concise decision-facing claim | cell ID plus source version | material conditions | independent test and result | supported / qualified / rejected / unresolved | named reviewer |
Keep descriptive, predictive, and causal statements distinct. A correlation or forecast is not evidence that one factor caused another.
### 6. Decision handoff
End with the answer to the original question, material limitations, rejected alternatives, follow-up work, and a link to the accepted output. State which cells form the release path and which are retained only as exploratory history.
Use AI at bounded points
AI can help draft code, suggest tests, explain unfamiliar functions, generate synthetic edge cases, or critique the claim register. The analyst remains responsible for the source authority, transformation logic, execution environment, calculations, and interpretation.
For each material AI contribution, preserve enough context to reproduce the decision: the task given, relevant model or tool identifier when available, the proposed change, what the analyst checked, and whether it was accepted, edited, or rejected. Do not treat a plausible explanation as evidence that code ran correctly.
Useful challenge prompts include:
Run every suggested test yourself against approved data and inspect the result.
- “List assumptions in this aggregation that could change the decision.”
- “Construct a small table that would fail this join or denominator rule.”
- “Identify which sentence in the summary is not directly supported by a cell.”
- “Propose an alternative explanation, then specify the data needed to distinguish it.”
Worked notebook: renewal analysis
A synthetic subscription team asks whether a lower renewal rate in the small-business segment warrants a retention intervention. The analyst receives account, contract, and renewal-event extracts.
The input manifest records three snapshots and notes that renewal events may arrive up to seven days late. The contract defines the population as contracts reaching an expiry date in the quarter, not accounts active during the quarter. That distinction prevents the denominator from expanding with customers who had no renewal opportunity.
The transformation ledger exposes two problems. First, an account can have several contracts, so joining renewal events at account level duplicates outcomes. A grain assertion fails, and the analyst repairs the join to contract ID. Second, contracts still inside the seven-day reporting window are immature. They remain visible in a completeness table but are excluded from the main comparison under a recorded rule.
An AI assistant proposes imputing missing segment labels from company size. The analyst rejects the suggestion because company-size data are stale and the imputation would move records between the compared groups. The notebook instead reports unknown segment as a separate category and shows its share.
The main result is a 6.2 percentage-point descriptive difference. A challenge cell recalculates the rate from a ten-row fixture, checks the result under two timing windows, and shows that the difference narrows to 2.1 points after excluding immature contracts. The claim register therefore rejects “small-business customers are churning because onboarding is weak” and supports only: “In the mature quarterly cohort, recorded small-business renewal is lower; the available data do not identify the cause.”
The handoff recommends a targeted investigation rather than an automated intervention. It names the missing evidence—onboarding exposure, product usage, cancellation reasons, and contract changes—and retains the exact cohort extract, code revision, environment file, test output, and reviewer disposition.
Acceptance and restart test
Before release:
Hold the notebook when inputs cannot be versioned, hidden state changes a result, a material join or denominator lacks a test, sensitive data are handled outside approved controls, a claim cannot be traced, or no reviewer can rerun the accepted path. A passing restart test shows reproducibility for the frozen packet, not that future data or another decision will produce the same conclusion.
Two Prune can use a notebook task in a realistic AI-enabled work assessment and evaluate how a participant frames the analysis, navigates evidence, uses AI, verifies calculations, manages uncertainty, and communicates a decision. It does not operate the customer’s analytical pipeline or make employment decisions.
- restart the runtime and clear all outputs;
- run the accepted path from top to bottom in the frozen environment;
- confirm the input versions and expected record-count assertions;
- fail the run deliberately on one invalid input and inspect the message;
- reproduce at least one headline value independently;
- have a reviewer trace one claim from the handoff back to source records;
- confirm that rendered outputs contain the data-as-of time and material limitations.
Sources
Jupyter nbformat: The notebook file format
- UK Government Analysis Function: Reproducible Analytical Pipelines
Sources: [1] [2]
Sources
- 1.Jupyter nbformat: The notebook file format · nbformat.readthedocs.io
- 2.UK Government Analysis Function: Reproducible Analytical Pipelines · analysisfunction.civilservice.gov.uk