Practical guide
What evidence is needed to measure AI-assisted decision quality?
Measuring AI-assisted decision quality requires a reconstructable decision record. A final outcome alone is too late and too noisy; a polished recommendation alone hides how evidence was selected, how AI changed the work, and whether review controls operated. The evidence packet must connect the decision context, source record, AI and human transformations, challenge process, disposition, and.
Measuring AI-assisted decision quality requires a reconstructable decision record. A final outcome alone is too late and too noisy; a polished recommendation alone hides how evidence was selected, how AI changed the work, and whether review controls operated. The evidence packet must connect the decision context, source record, AI and human transformations, challenge process, disposition, and later outcome without treating correlation as causation. The practical output is a versioned evidence inventory that lets an independent reviewer answer: what was decided, using which information, under which constraints, what AI contributed, what the person verified or changed, what remained unknown, and what happened next?
Define the measurement boundary
Freeze the construct before collecting convenient telemetry. Specify the decision type, decision owner, affected people, available alternatives, authority limit, time and information cutoff, consequence of error, expected evidence standard, allowed AI role, required review, and observation window.
Separate six questions:
These are evidence dimensions, not a universal scoring formula. Weighting and acceptance conditions must reflect the decision's context and consequence.
- Was the problem framed correctly?
- Was relevant evidence found, weighed, and cited?
- Was AI used within the stated role and inspectably?
- Were material claims, calculations, conflicts, and uncertainty checked?
- Did the rationale address alternatives and tradeoffs?
- Did the final artifact enable an authorized person to act?
Use the RECORD evidence architecture
RECORD keeps primary records separate from interpretations and later outcomes.
### R — Register the decision and version
Assign a decision ID. Record the question, owner, contributors, alternatives, constraints, cutoff time, materiality threshold, AI configuration, and exact versions of the final artifact and source set. Preserve revisions rather than overwriting them. A later reviewer must know what information was actually available at the decision point.
### E — Evidence source map
For every material claim, identify the primary source, source owner, date, location anchor, authority, known limitation, and whether it was available before the cutoff. Include evidence that contradicts the preferred option and sources considered but rejected. Record missing, unreadable, stale, or inaccessible evidence explicitly.
The NIST AI Risk Management Framework Measure playbook says measurement choices should follow purpose, audience, and context; calls for methods to detect risks and errors; and recommends documenting metrics, limits, unmeasured characteristics, test results, and corrective actions. Use that discipline to explain the evidence plan. It does not prescribe a single metric for decision quality or validate a local measure. NIST AI RMF Measure playbook
### C — Capture AI and human transformations
Preserve the material inputs, prompts or instructions, tool outputs used, retrieval results, calculations, human edits, rejected suggestions, and final synthesis. Avoid collecting irrelevant personal content; the evidence plan itself needs privacy and access controls.
Classify each material contribution as source retrieval, extraction, calculation, comparison, generation, critique, or formatting. Then record who verified it and how. “Human reviewed” is not enough: specify the source check, recalculation, counterexample, independent comparison, or domain review performed.
### O — Observe challenge, control, and authorization
Collect the challenge record: contradictory evidence introduced, assumptions tested, alternative explanations considered, reviewer comments, disagreements, escalations, and changes made. Record whether the person could pause, reject, or revise the AI contribution and whether time, access, and incentives made that control practical.
The US Government Accountability Office's AI accountability framework organizes review around governance, data, performance, and monitoring and supplies questions for managers and assessors. It can help structure control and monitoring evidence, but using the framework does not amount to an external approval of a workflow. GAO AI Accountability Framework
### R — Resolve missingness and confounds
Create a missingness log. For each absent item, state why it is missing, whether absence is systematic, whose work is excluded, how it affects interpretation, and whether the decision should be held. Missing AI traces, unrecorded offline discussion, unavailable rejected alternatives, or only retaining successful cases can each distort the measure.
Pre-register important confounds: case difficulty, participant experience, tool access, reviewer severity, source completeness, time pressure, changing policies, team support, and selection into AI use. Where feasible, compare like cases, calibrate reviewers, sample exceptions, and report distributions rather than one average.
### D — Distinguish disposition from downstream outcome
Record the decision disposition at the cutoff, then observe outcomes on a separate timeline. Later results can inform learning, but they may be shaped by implementation quality, external events, chance, or information that was not available earlier. A sound decision can have a poor outcome; a weak process can have a lucky outcome.
HM Treasury's Green Book distinguishes appraisal from monitoring and evaluation and frames evidence around objectives, options, costs, benefits, risks, and later learning. Its rules apply to relevant public-sector proposals; here, the useful principle is to keep ex-ante decision quality distinct from ex-post outcome evaluation. HM Treasury: The Green Book 2026
Sources: [1] [2] [3]
Minimum evidence inventory
| Evidence object | Primary contents | Verification | Common gap | | --- | --- | --- | --- | | Decision register | question, owner, alternatives, cutoff, authority | owner confirmation | decision rewritten after result | | Frozen source set | source, date, anchor, authority, limitation | source-location check | only supporting sources kept | | AI contribution log | material inputs, outputs used, configuration | trace-to-artifact check | full chat retained but use unclear | | Human action log | edits, rejections, calculations, escalations | reviewer reconstruction | “reviewed” without action detail | | Rationale map | claim, source, assumption, alternative, uncertainty | independent challenge | conclusion without rejected option | | Control record | reviewer, test, finding, response, authorization | sampled operating evidence | policy exists but did not operate | | Missingness log | missing item, cause, bias risk, treatment | completeness audit | absent cases silently dropped | | Outcome record | later event, window, implementation context | separate evaluation | outcome treated as process proof |
Verification method
Select a risk-based sample that includes ordinary cases, exceptions, low-performing cases, and cases with reviewer disagreement. Give a second reviewer the frozen packet without the original conclusion and ask them to reconstruct the decision, locate the controlling evidence, identify the AI contribution, repeat material calculations, and explain the disposition.
Report reconstruction success, source-anchor accuracy, calculation agreement, contradiction detection, escalation correctness, reviewer agreement, missingness, and review effort. Use distributions and case notes. If reviewers disagree, examine the rubric, evidence, and authority boundary before averaging their scores.
Acceptance requires enough evidence to reconstruct the decision and distinguish observed facts from interpretation. It does not require collecting every click or keystroke. Excess telemetry can create privacy risk without improving measurement.
Worked artifact: warehouse staffing schedule decision
A regional distribution center must choose one of three staffing schedules for a four-week seasonal peak. A planner may use AI to compare demand forecasts, absence patterns, skill coverage, overtime rules, and agency capacity. The decision owner is the site operations manager; employee relations and safety reviewers have defined review roles.
The frozen source set contains forecast versions, prior-week volumes, role qualifications, approved overtime rules, absence ranges, agency lead times, and a safety constraint on consecutive shifts. One spreadsheet labels volume in cases, while a planning note uses pallets. The evidence packet preserves that conflict.
The AI proposes schedule B and states that it has the lowest expected cost. The human planner checks the calculation and discovers that the model compared cases with pallets and omitted agency induction time. The planner rejects the cost claim, rebuilds the comparison with a documented conversion range, and keeps schedules A and B open. A reviewer challenges whether the highest forecast scenario would breach the consecutive-shift constraint.
The final disposition selects schedule A for week one and sets a review gate before weeks two through four. The rationale cites the controlling sources, records the conversion uncertainty, explains why B was not selected, and assigns an owner to obtain a reconciled forecast. The outcome record begins later and tracks coverage, overtime, safety exceptions, forecast error, and manager overrides separately.
The evidence supports a finding that the planner detected a unit conflict, corrected the AI-assisted comparison, considered alternatives, and communicated uncertainty. It does not establish that AI caused the final operational outcome or that the same process will generalize to other sites or seasons.
Interpretation limits and failure conditions
Hold interpretation when the decision question changed after the fact, source versions cannot be recovered, only successful cases remain, material offline work is missing, reviewers are uncalibrated, outcome knowledge contaminated the process review, or the AI contribution cannot be separated from the human artifact.
Avoid proxies that collapse the construct. Prompt count is not orchestration quality. Citation count is not evidence quality. Agreement with the AI is not correctness. Confidence is not verification. Cycle time is not decision quality. A final outcome is not a complete measure of the reasoning available at the cutoff.
Report the population, case mix, tool configuration, source coverage, observation window, missingness, reviewer disagreement, control failures, and transfer limits. Reopen the evidence plan when the decision, source environment, model, interface, workflow, or consequence changes.
Two Prune boundary
Two Prune can run realistic work simulations and preserve evidence about observed participant performance across context framing, evidence navigation, AI orchestration, verification and risk control, judgment and synthesis, and communication. It does not operate a customer's live decision process, establish causal business impact, certify controls, provide a universal decision-quality measure, or make employment decisions. Customers retain responsibility for decision authority, tools, data, measurement design, interpretation, and use of assessment evidence.
Sources
- 1.NIST AI RMF Measure playbook · National Institute of Standards and Technology
- 2.GAO AI Accountability Framework · U.S. Government Accountability Office
- 3.HM Treasury: The Green Book 2026 · UK Government