How to develop and assess causal reasoning in AI-enabled work

Separate change from attribution

HM Treasury's 2026 Magenta Book distinguishes observing an outcome from determining whether an intervention caused it. It describes counterfactual approaches, theory-based methods, and the need to test alternative explanations. The CDC Program Evaluation Framework 2024 likewise describes impact evaluation as assessing causal impact relative to a counterfactual condition in which the programme did not occur.

This creates a practical boundary for everyday decisions. A before-and-after difference shows change. A participant comment suggests a mechanism. A correlation identifies a relationship worth investigating. None automatically establishes what caused the outcome. The worker should say what the evidence design permits and what remains uncertain.

Build a CAUSAL claim card

Use one CAUSAL card for each material causal statement. The card forces the claim and its evidence design into the same view.

| Field | Question | Evidence to retain | | --- | --- | --- | | Claim | What action is said to cause which outcome, for whom, and when? | One bounded causal sentence | | Assumed mechanism | Through what sequence could the action produce the outcome? | A short causal pathway with observable steps | | Unexposed comparison | What would likely have happened without the action? | Control, matched group, baseline trend, phased rollout, or explicit absence | | Sources of difference | How could selection, timing, measurement, attrition, or concurrent events distort the comparison? | Rival explanations and directional bias | | Analysis fit | What design and measure can answer the question? | Chosen method, assumptions, and checks | | Limits | What narrower language does the evidence support? | Effect, contribution, association, observation, or unresolved |

The card begins with a claim, not a preferred method. "The programme improved quality" is too broad. A usable version might ask whether a defined review step reduced a specified error type during a fixed period for teams subject to the same reporting rule. The mechanism and comparison can then be designed around that proposition.

Draw the mechanism before testing it

Write the expected sequence from action to outcome. If a new review checklist is meant to reduce unsupported claims, the proposed path may be: checklist introduced, reviewers apply named checks, unsupported statements are detected before release, revisions occur, and released documents contain fewer such statements.

Each arrow needs an observable implication. Adoption records can show whether the checklist was used. Review comments can show whether the named checks found problems. Version history can show whether corrections occurred. A final error count alone cannot reveal which step operated.

AI can help enumerate mechanisms and rival explanations, but its list is not evidence. The worker should use process records, domain knowledge, and the intervention's actual design to decide which pathways are plausible and testable.

Choose a comparison that matches the decision

The strongest feasible comparison depends on how the action was assigned and what data exist. Random assignment can create comparable groups when appropriate and implemented correctly. A phased rollout may allow comparison across timing. A matched comparison or interrupted time series can be useful when its assumptions are credible. Theory-based approaches can test mechanisms and alternatives when a robust counterfactual is not feasible.

The What Works Clearinghouse's current Version 5.0 materials set separate standards for randomized, quasi-experimental, regression-discontinuity, and single-case designs. The lesson for workplace analysis is not that every operational choice requires a formal trial. It is that the method determines which causal language is defensible. A convenience comparison should not be described as if it had the properties of random assignment.

Worked example: a review-step pilot

Imagine a proposal team piloting a source-check step for four weeks. This is a synthetic example and does not describe a Two Prune customer, product result, or live bid.

After the pilot, the share of reviewed statements that require source correction is lower than in the prior four weeks. An AI summary concludes that the source-check step caused the improvement. The analyst creates a CAUSAL card before accepting that wording.

The bounded claim is that the added step contributed to fewer source-correction findings in comparable proposal reviews during the pilot. The mechanism predicts that reviewers used the checklist, found unsupported statements earlier, and prompted revisions before the formal review.

The comparison is imperfect. The pilot work involved different proposal types, two experienced reviewers joined during week two, and the definition of a correction changed in the new checklist. The analyst reconstructs the same error definition across both periods, stratifies records by proposal type, checks reviewer assignment, and examines weekly trends before the intervention.

Process evidence shows the checklist was completed in most pilot reviews and version histories contain corrections linked to its named checks. However, the staffing change remains a credible alternative explanation, and no concurrent comparison group exists. The analyst revises the conclusion: "The pilot period had fewer comparable source-correction findings, and the review trail is consistent with the checklist contributing to earlier correction. The before-and-after design cannot isolate its effect from reviewer mix and other period changes."

The artifact contains the causal pathway, comparison definition, reconstructed measure, rival-explanation table, process evidence, analysis output, and calibrated statement. It helps decide whether to run a broader phased test without overstating what the pilot established.

Practice with rival explanations

Give a learner a small dataset showing an outcome change after an intervention. Ask for at least four alternative explanations grouped by selection, measurement, timing, concurrent events, and natural variation. For each alternative, require one observable check and a statement of what result would weaken or strengthen it.

Then change the design. Add a comparison group, show a pre-intervention trend, or reveal that the measurement rule changed. The learner should update the causal statement rather than defend the first answer. This tests whether the worker treats causal reasoning as revision under evidence rather than a persuasive story.

A mechanism drill can use qualitative material. Give the learner implementation notes, interviews, and version histories. Ask which observations would be expected if the proposed mechanism operated and which would appear under a rival mechanism. The task is to discriminate between explanations, not to collect supportive anecdotes.

Assess the reasoning from the work trail

A realistic assessment can combine a brief, a before-and-after metric, operational records, a policy change, participant comments, and one concurrent event. Ask the participant for a decision note and a completed CAUSAL card.

Review whether the participant:

Do not score causal reasoning by statistical complexity or by whether the participant recommends continuation. A simple design can support useful reasoning when assumptions and limits are explicit. A sophisticated model can still be misleading when its comparison is unsuitable or its inputs changed. Retain the claim card, calculation or code, measure definitions, comparison rationale, alternative-explanation log, and revision history.

• defines the cause, outcome, population, and timing;
• distinguishes an observed change from attribution;
• makes the causal pathway explicit and testable;
• chooses a comparison suited to assignment and available data;
• checks measurement consistency, selection, attrition, timing, and concurrent changes;
• seeks evidence that could disconfirm the preferred explanation;
• narrows causal language when the design cannot isolate an effect.

Verify the final causal statement

Read every sentence containing words such as caused, led to, improved, reduced, because, impact, or effect. For each one, identify the comparison and mechanism that support it. Confirm that outcome definitions are consistent, timing is plausible, groups are comparable under the design, and known concurrent changes were examined.

Run a claim-downshift test. Rewrite the statement at four levels: causal effect, contribution, association, and observation. Choose the strongest level the evidence actually supports. Record the assumptions that would need to hold for the stronger level and avoid hiding them in a footnote.

Causal reasoning supports better decisions about what to test, continue, adapt, or stop. It does not turn observational data into proof or establish transfer beyond the studied conditions. In a Two Prune assessment, the claim card and verification trail can contribute evidence about Judgment & Synthesis and Verification & Risk Control within the simulation. They are task-bounded assessment evidence, not a prediction of future performance or an automated employment decision.

Sources

HM Treasury, Magenta Book: Central Government guidance on evaluation

• CDC Program Evaluation Framework, 2024
• What Works Clearinghouse, Handbooks and Other Resources

Sources: [1] [2] [3]