Practical guide
What does it mean to measure AI-assisted decision quality?
Measuring AI-assisted decision quality means evaluating whether a bounded decision was framed, evidenced, challenged, authorised, and made under uncertainty in a way appropriate to its stakes. It is not a score for how polished the AI output appears, whether the reviewer agrees with the choice, or whether the eventual outcome happened to be favourable. The construct must freeze what was knowable.
Measuring AI-assisted decision quality means evaluating whether a bounded decision was framed, evidenced, challenged, authorised, and made under uncertainty in a way appropriate to its stakes. It is not a score for how polished the AI output appears, whether the reviewer agrees with the choice, or whether the eventual outcome happened to be favourable. The construct must freeze what was knowable at the decision time, show where AI contributed, preserve what people checked or rejected, and record later outcomes separately. Use the BOUND definition: Boundary, Options, Underlying evidence, Negative challenge, and Decision ownership.
B — Boundary: define the decision unit and information cutoff
Record one decision, its owner, date, objective, audience, stakes, reversibility, constraints, required approvals, affected people, and the information available by the cutoff. Define the decision family narrowly enough that evidence and scoring anchors make sense: selecting a supplier for a bounded pilot is different from approving a long-term enterprise contract.
Separate the decision process from the later result. A well-reasoned reversible choice can face an adverse external event; a poorly supported choice can benefit from luck. Later evidence should update future decisions, but it should not be inserted into the original record as though the decision maker knew it.
The 2026 UK Green Book calls for clear objectives, broad option generation, documented reasons, transparent assumptions, risk and uncertainty analysis, and evidence-informed judgment. It is government appraisal guidance rather than a universal scoring model, but it supports freezing the rationale and alternatives available to a decision maker. The Green Book 2026
Sources: [1]
O — Options: test whether there was a genuine choice
Measure whether the decision considered a sufficient set of viable alternatives. Include business as usual, a minimum change, a narrower or more reversible option, and stopping where relevant. Record why options were included, rejected, or deferred and whether the AI system generated, summarized, or ranked them.
Option count alone is not quality. Ten superficial variants may represent one underlying path, while two materially different choices may be sufficient for a low-stakes reversible decision. Define observable anchors such as: alternatives address the objective, critical dependencies are visible, comparisons use consistent definitions, and rejected choices retain a reason.
Beware automation anchoring. If reviewers see the AI recommendation first, they may evaluate alternatives around its frame. Preserve independent option generation or a challenge step when the consequence warrants it.
U — Underlying evidence: trace claims, calculations, and AI contribution
For each material premise, record the source, date, relevance, limitation, and owner. Recalculate important numbers and check units, denominators, time periods, exclusions, assumptions, and sensitivity. Distinguish observed facts, forecasts, estimates, expert judgment, and missing evidence.
Map the AI contribution: retrieval, summarization, option generation, calculation, scenario analysis, critique, or drafting. Then record what a person verified, changed, rejected, or escalated. An AI transcript is not automatically useful evidence; retain only the proportionate records needed to understand the decision.
NIST's AI RMF Core calls for context, intended use, roles, human oversight, testing, documentation, risk measurement, and interpretation of AI outputs within their context. These outcomes support tracing the human-AI decision configuration without claiming that framework use validates the decision. NIST AI RMF Core
Missing workflow evidence should reduce confidence in the measure. Do not convert an absent trace into proof of poor judgment, and do not treat a complete log as proof that the reasoning was sound.
Sources: [2]
N — Negative challenge: expose uncertainty and rival cases
Measure whether the process looked for information that could change the choice. Record uncertainty ranges, dependencies, possible unintended consequences, disconfirming evidence, stakeholder objections, scenarios, and the condition at which the preferred option should switch.
The UK Orange Book says risk management should support objectives and decision-making, make principal risks transparent, assess alternatives and scenarios, identify dependencies and bias, assign accountable owners, and monitor treatment. Use those principles to define a challenge artifact; they do not establish that a particular residual risk is acceptable. Orange Book: Management of Risk
For a consequential decision, require an independent challenge with source citations. Preserve disagreement when it reflects a real ambiguity. A review is weak when it asks only whether the recommendation sounds reasonable, or when the same person creates, challenges, approves, and scores the decision without a recorded justification.
Sources: [3]
A construct card for AI-assisted decision quality
| Field | Required definition | | --- | --- | | Construct | quality of the bounded decision process given the evidence and uncertainty available at the cutoff | | Unit | one decision within a named family and stakes class | | Population | decisions eligible under the stated workflow, not every AI interaction | | Dimensions | boundary, options, underlying evidence, negative challenge, decision ownership | | Primary evidence | decision record, source set, calculations, AI-contribution map, reviewer notes, approval and action | | Missingness | explicit missing, not automatically zero or excluded | | Comparison | only sufficiently similar decision families, stakes, information conditions, and review standards | | Later outcomes | linked but reported separately from ex-ante decision quality | | Confidence | based on evidence coverage, reviewer calibration, and unresolved ambiguity | | Prohibited inference | general capability, causal impact, future performance, or automated employment decision |
Each dimension needs observable anchors for the decision family. A procurement choice may require comparable price and service assumptions; an incident response may prioritize timeliness, containment, evidence preservation, and escalation. A universal weighted score can hide a critical failure, so define non-compensable conditions separately.
Measurement sources, confounds, and missingness
Use a combination of the decision artifact, source documents, calculation checks, version history, AI-use evidence where proportionate, reviewer observations, approvals, and later follow-up. Self-report can explain intent and barriers but should not stand alone as process evidence.
Common confounds include decision difficulty, stakes, time pressure, source quality, tool configuration, prior expertise, reviewer leniency, and hindsight. Record these before comparing decisions. Calibrate reviewers with paired cases that differ on one material dimension, require evidence citations, and preserve disagreements that could change the disposition.
For missing evidence, choose and record one rule: repair collection, retain an unknown category, publish with a confidence limit, narrow the interpretation, or withhold the score. Do not assume an unobserved check did not occur, and do not assume that it did.
Worked construct: cold-chain carrier selection
This synthetic example concerns a regional operations group selecting a carrier for a six-month cold-chain pilot. Three carriers submit proposals. An AI assistant summarizes price schedules, route coverage, temperature-monitoring terms, incident procedures, and references.
The decision record fixes the cutoff at 30 June and defines the objective: select a reversible pilot option that can serve two routes while meeting stated monitoring and escalation requirements. Business as usual and a smaller one-route pilot remain alternatives. The stakes class is moderate because the pilot is bounded, but temperature-control failure could have material consequences.
BOUND produces this review:
| Dimension | Observed evidence | Limitation or challenge | Disposition | | --- | --- | --- | --- | | Boundary | objective, routes, pilot duration, approval, and stop conditions are explicit | one service-volume forecast remains uncertain | pass with sensitivity check | | Options | three carriers, business as usual, and one-route pilot compared | AI initially merges two carriers' premium-service terms | repair the comparison from source proposals | | Underlying evidence | prices recalculate and route coverage links to dated proposals | one reference is vendor-selected and not independent | qualify reference evidence | | Negative challenge | reviewer tests a 15% volume increase and a sensor outage scenario | no evidence yet for recovery time after a regional outage | retain as open condition and narrow pilot | | Decision ownership | operations recommends; quality and procurement review; authorised director approves | technology owner must confirm sensor export before start | approval conditional on export test |
The AI summary initially ranks Carrier A first because it treats an optional premium service as included. Source checking corrects the price and changes the preferred option to a one-route pilot with Carrier B. The director approves only after the sensor-export test, with a stop condition for missing temperature records and a 30-day review.
If Carrier B later experiences an unrelated road closure, that outcome does not retroactively prove the decision process was poor. The follow-up record separately compares forecast assumptions, actual monitoring performance, incidents, and whether the stop rule operated. Future calibration may change, but the original information cutoff remains intact.
Interpretation limits
A high-quality decision record supports a bounded interpretation: under the stated conditions, the observed process framed the decision, compared alternatives, traced evidence, challenged uncertainty, and assigned accountable action. It does not show that every relevant fact was available, the outcome will be favourable, the AI system caused the quality, the person has general decision capability, or the measure is reliable across other families and reviewers.
Two Prune boundary
Two Prune can assess observed decision work in a realistic simulation, including context framing, evidence navigation, AI orchestration, verification and risk control, judgment and synthesis, and communication. It can explain the evidence behind an assessment score. It does not validate a universal decision-quality construct, predict future workplace outcomes, certify risk controls, or make employment decisions. Customers define the decision family, stakes, rubric, qualified reviewers, operational authority, and permitted use of the evidence.
Sources
- 1.The Green Book 2026 · UK Government
- 2.NIST AI RMF Core · National Institute of Standards and Technology
- 3.Orange Book: Management of Risk · UK Government