Practical guide
Common mistakes when measuring AI-assisted decision quality
The most serious measurement mistake is treating a good outcome as proof of a good decision. AI-assisted decision quality concerns whether the decision process used relevant evidence, tested uncertainty, controlled consequential errors, and produced a justified choice at the time. Outcomes matter, but luck, delay, external events, and hidden interventions can separate the eventual result from.
The most serious measurement mistake is treating a good outcome as proof of a good decision. AI-assisted decision quality concerns whether the decision process used relevant evidence, tested uncertainty, controlled consequential errors, and produced a justified choice at the time. Outcomes matter, but luck, delay, external events, and hidden interventions can separate the eventual result from the quality of the original reasoning.
Define the construct before choosing a metric
Write a construct statement that can be challenged. For example:
> AI-assisted decision quality is the extent to which a person frames the decision, uses relevant and current evidence, tests AI-supported claims, considers alternatives and uncertainty, observes required controls, and communicates a justified choice appropriate to the decision context.
This definition separates process quality from speed, tool usage, confidence, writing polish, and the later outcome. Choose observable evidence for each element: sources opened, claims traced, calculations checked, alternatives compared, uncertainties recorded, escalation used, and the final rationale. Preserve missing evidence as missing rather than turning it into a low score.
NIST’s AI Risk Management Framework calls for contextual evaluation, documented roles, defined human oversight, representative testing, and ongoing measurement and monitoring. It also treats risk management as iterative rather than a one-time checklist. These are useful design prompts, not a ready-made measure of decision quality. NIST AI RMF Core
Sources: [1]
Run an interpretation-risk audit
Before reporting results, test the analysis against seven recurring traps.
### 1. Outcome hindsight
Judging the process after seeing the result rewards lucky choices and punishes sound choices followed by an adverse external event. Score the contemporaneous evidence and rationale separately from the later outcome. Where possible, have reviewers assess the decision record before outcome information is revealed.
### 2. Proxy substitution
Prompt count, AI usage time, completion speed, self-confidence, or prose quality may be easy to collect, but none is the construct. A participant can use many prompts without checking a source, or write elegantly around an unsupported claim. State what each proxy can and cannot represent.
### 3. Selected and missing cases
Results can look stronger when incomplete attempts, escalations, tool failures, or difficult cases disappear from the denominator. Predefine the eligible set and report included, excluded, incomplete, and missing cases with reasons. Never silently treat “not observed” as “did not occur.”
### 4. Reviewer contamination
A reviewer who sees the participant’s identity, overall score, fluent final answer, AI transcript, or outcome may let that information influence unrelated judgments. Use a scoring order, evidence anchors, and masking where practical. Adjudicate disagreements that cross a material threshold instead of averaging away the reason.
### 5. Aggregation that hides a critical failure
An average can conceal a fabricated source, sensitive-data disclosure, unapproved automation, or failure to escalate a high-consequence uncertainty. Define non-compensatory conditions separately. A strong communication score should not erase a failed evidence control.
### 6. Causal overreach
An association between AI use and a result does not show that AI caused the result. Team selection, task difficulty, coaching, tool access, baseline skill, and concurrent process changes may explain the difference. The UK government’s Magenta Book Annex notes that evaluation must address whether an intervention caused an impact, how context influenced outcomes, and how far findings generalize; method choice determines what attribution is supportable. Magenta Book Annex A
### 7. Transfer beyond the evidence
A result from one task, domain, language, tool configuration, or support condition does not automatically transfer elsewhere. Report the tested population and context. Treat a new workflow as a new inference problem, especially when sources, stakes, or human review change.
Sources: [2]
Worked example: an inventory replenishment recommendation
A synthetic assessment asks 20 participants to recommend whether to increase replenishment for three products. The packet contains demand history, a supplier notice, current inventory, margin data, and an older promotional forecast. An AI tool is available. The intended evidence includes source selection, calculation checks, treatment of the stale forecast, uncertainty, alternatives, and the final recommendation.
The initial dashboard reports: 16 of 20 recommendations were followed by acceptable stock levels, AI users finished 18% faster, and average reviewer score was 82%. Those figures appear positive but are not yet interpretable.
The audit finds:
| Trap | Observed problem | Correction | | --- | --- | --- | | Outcome hindsight | Reviewers saw later stock levels before scoring rationale | Rescore the contemporaneous record with outcomes masked; report outcomes separately | | Proxy substitution | Faster completion is labelled “better judgment” | Relabel as elapsed time and pair with verification and error evidence | | Missing cases | Two tool failures were removed from the denominator | Restore all 20 eligible cases and report the failure mode | | Reviewer contamination | Polished recommendations received higher evidence scores despite missing citations | Score source traceability before presentation quality | | Hidden critical failure | One high total included a fabricated supplier lead time | Flag unsupported external facts as a separate critical condition | | Causal overreach | AI users had more prior planning experience | Describe the result as observational; do not attribute the difference to AI | | Transfer | All cases used low-volatility products and one supplier | Limit interpretation to that task design and specify a higher-volatility test |
After correction, the useful conclusion is narrower: most participants completed the synthetic task, but source-version handling and unsupported supplier claims require repair; elapsed-time differences cannot be attributed to AI because assignment and baseline experience were not controlled; no claim about live inventory outcomes is supported.
Build an interpretation-risk register
Keep a compact register beside every decision-quality report.
| Field | Question | | --- | --- | | Construct | What capability is being inferred, and what is explicitly outside it? | | Evidence unit | What task, decision, person, and time window produced the record? | | Denominator | Which cases were eligible, included, incomplete, or excluded? | | Missingness | Why is evidence absent, and could absence relate to performance? | | Confounds | What else could explain the observed difference? | | Reviewer exposure | What identity, outcome, transcript, or prior score could influence judgment? | | Critical conditions | Which failures cannot be offset by strengths elsewhere? | | Transfer boundary | To which tasks, tools, settings, and populations does the evidence apply? | | Claim level | Is the statement descriptive, comparative, causal, or predictive? | | Next test | What evidence would most reduce the remaining uncertainty? |
GAO’s AI Accountability Framework organizes practices around governance, data, performance, and monitoring. Its emphasis on traceability, performance assessment, and continued monitoring is a useful cross-check for this register, while the organization remains responsible for choosing methods suited to its context. GAO AI Accountability Framework
Sources: [3]
Reporting rules that prevent overstatement
Report distributions and critical conditions, not only averages. Attach denominators to rates. Separate assessment evidence, live-process evidence, and downstream outcomes. Name confounds and missingness beside the finding they affect. Use “observed in this assessment” when that is the evidence boundary. If a comparison is not randomized or otherwise designed for attribution, keep the claim descriptive.
For early or small benchmark sets, label percentiles as directional or unavailable and show the comparison group. A precise-looking percentile does not repair an unrepresentative reference set.
How this applies to Two Prune
Two Prune can observe performance inside a realistic work simulation, including evidence navigation, AI orchestration, verification, judgment, revisions, and the final deliverable. Its report should explain the evidence supporting an assessment score and where confidence is limited. It does not establish how a participant will perform in every future context, prove an operational outcome, or make an employment decision.
Sources and limits
These sources support disciplined evaluation questions. They do not validate this interpretation-risk audit or any particular decision-quality score. The worked example is synthetic.
- NIST AI RMF Core — contextual evaluation, roles, oversight, measurement, and monitoring considerations.
- Magenta Book Annex A: analytical methods — official guidance on causal questions, context, generalisability, and evaluation methods.
- GAO AI Accountability Framework — governance, data, performance, and monitoring practices.
Sources: [1] [2] [3]
Sources
- 1.NIST AI RMF Core · National Institute of Standards and Technology
- 2.Magenta Book Annex A · UK Government
- 3.GAO AI Accountability Framework · U.S. Government Accountability Office