Practical guide

A practical method for measuring AI-assisted decision quality

A decision-quality method should measure what was knowable and done at the decision point, not reward a polished AI response or a lucky later outcome. The protocol must define the decision population, sample representative cases, preserve source and AI-human evidence, use anchored independent review, quantify missingness and disagreement, and report where the result can and cannot transfer. Use.

By Two Prune

A decision-quality method should measure what was knowable and done at the decision point, not reward a polished AI response or a lucky later outcome. The protocol must define the decision population, sample representative cases, preserve source and AI-human evidence, use anchored independent review, quantify missingness and disagreement, and report where the result can and cannot transfer. Use PAIR: Population and construct, Assessment-case design, Independent rating, and Reporting with uncertainty.

P — Population and construct specification

Define the exact decision type, decision owner, affected people, stakes, alternatives, authority limit, information cutoff, allowed AI role, required verification, and observation window. Describe the eligible population and the cases excluded. A method for procurement renewals does not automatically apply to safety incidents, staffing schedules, or customer remedies.

Write a construct table before collecting data:

| Dimension | Observable evidence | Failure example | Not evidence by itself | | --- | --- | --- | --- | | Framing | decision, audience, constraints, success criteria | solves a different problem | long prompt | | Evidence | controlling sources, anchors, conflicts, limits | relies on stale or weak source | citation count | | Alternatives | genuine options and tradeoffs | preferred route treated as inevitable | extra prose | | AI use | inspectable contribution and human control | suggestion accepted without scrutiny | tool usage | | Verification | claim, calculation, conflict, and uncertainty checks | material issue left untested | confidence statement | | Judgment | rationale connects evidence to authorized action | conclusion outruns evidence | agreement with reviewer | | Delivery | artifact enables the intended decision | authority or next step unclear | formatting quality |

These dimensions are a measurement model, not a universal weighting scheme. Set weights and minimum conditions according to consequence. Some failures should be non-compensatory: a high average should not erase fabricated evidence or an unhandled prohibited condition.

NIST's AI Risk Management Framework Core calls for experimental design, construct validation, representative evaluation conditions, documented test sets and metrics, independent input, uncertainty, and limits on generalization. It supports documenting why a method fits its context; using the framework does not validate the resulting score. NIST AI RMF Core

Sources: [1]

A — Assessment-case and sampling design

Build a case blueprint from the real decision population. Stratify by difficulty, source completeness, consequence, exception type, and the kinds of conflict the decision maker must resolve. Include ordinary, edge, and adverse cases. Freeze the source pack, policy versions, tool configuration, and information cutoff.

For each case, create an evidence key that identifies the controlling sources, defensible alternatives, material uncertainty, expected verification actions, escalation conditions, and unacceptable claims. The key guides reviewers; it should not require one stylistic answer or a single preferred choice when several choices are defensible.

Choose a sample large and varied enough for the decision the measurement must support. Record selection, exclusions, unavailable cases, protocol deviations, and who is missing. Do not select only successful AI interactions or balance groups after seeing the results. If the sample is small, report case-level results and uncertainty instead of precision the design cannot support.

The 2026 Magenta Book distinguishes evaluation questions, approaches, methods, and data collection; it emphasizes early design, proportionate methods, quality assurance, reproducibility, limitations, ethics, sampling bias, and measurement error. It applies to UK government evaluation, but these principles are useful for a transparent measurement protocol. HM Treasury: Magenta Book 2026

Sources: [2]

I — Independent rating, calibration, and adjudication

Create behavioral anchors for each score. A three-level anchor can be sufficient:

Add separate prohibited-condition flags for fabricated sources, material calculation error, hidden conflict, unhandled high-consequence uncertainty, unauthorized decision, or missing evidence represented as fact.

Calibrate reviewers with at least three paired cases: clear failure, clear pass, and genuine ambiguity. Reviewers score independently, explain evidence anchors, compare disagreements, and revise the guide before live scoring. Preserve the calibration versions and do not tune anchors solely to force agreement.

During scoring, blind reviewers to irrelevant identity and later outcome information where feasible. Double-score a risk-selected sample and all prohibited-condition cases. Record raw ratings, evidence anchors, disagreement, adjudication, and guide version. Agreement statistics can describe consistency, but a high coefficient does not show that the construct is correct or that both reviewers interpreted the evidence well.

Quality assurance should attempt reconstruction: can another reviewer locate the controlling source, reproduce the material calculation, identify the AI contribution, and explain the disposition from the packet? The Magenta Book recommends independent reproduction or sampled checks where appropriate and clear communication of limitations.

  • 0 — absent or materially unsafe: required evidence or control is missing, contradicted, invented, or outside authority;
  • 1 — partial: relevant evidence is present but incomplete, weakly verified, or poorly connected to the decision;
  • 2 — sufficient for this bounded case: the artifact uses controlling evidence, handles conflicts and uncertainty, and supports an authorized action.

R — Reporting, uncertainty, and transfer limits

Report more than a mean score. Include:

Keep decision-process quality separate from later outcome. Outcomes belong on a second timeline with implementation context and external events. Do not infer that AI caused a result merely because it appeared in the workflow.

GAO's AI Accountability Framework organizes evidence around governance, data, performance, and monitoring and recommends traceability, documentation, assessment, and corrective action. Use it to test whether the measurement packet supports accountability; it does not prescribe a universal decision-quality score. GAO AI Accountability Framework

Define missingness rules before analysis. If source versions, material AI traces, offline decisions, reviewer evidence, or failed cases are absent, state whether the case can be partially scored, must be held, or is excluded with a sensitivity analysis. Never convert missing evidence into a low participant score without support.

Reopen the method when the decision type, source system, AI configuration, interface, population, rubric, reviewer training, consequence, or use of scores changes. A result transfers only as far as the case design and evidence support.

  • case counts by difficulty and consequence;
  • score distributions by dimension;
  • prohibited-condition frequency and examples;
  • missing evidence and protocol deviations;
  • reviewer disagreement and adjudication rate;
  • review effort and evidence reconstruction success;
  • differences by case type, source completeness, and tool configuration;
  • the scope within which the result may be interpreted.

Sources: [3]

Copyable method card

| Field | Required entry | | --- | --- | | Use decision | What action the measurement will inform | | Population | Eligible decisions, people, settings, exclusions | | Construct | Dimensions, weights, minimums, prohibited conditions | | Cases | Sampling strata, difficulty, consequence, source completeness | | Evidence key | Controlling sources, alternatives, uncertainty, escalation | | AI boundary | Allowed contribution, configuration, records retained | | Review | Anchors, calibration set, double-score rule, adjudication | | Missingness | Partial-score, hold, exclusion, sensitivity rules | | Analysis | Distributions, agreement, exceptions, reconstruction | | Transfer | Supported scope, unsupported uses, change triggers |

Worked method: software renewal decision

An organization wants to measure how well analysts make AI-assisted recommendations on recurring software renewals. The population is contracts under a defined value threshold with usage data, service incidents, security reviews, and a named business owner. High-risk regulated systems are excluded because they require a different decision process.

The blueprint creates 18 synthetic cases across low, medium, and high ambiguity. Every case includes a current agreement, usage extract, incident history, security status, owner comments, pricing options, and decision deadline. Adverse cases include an auto-renewal date hidden in an amendment, active users counted twice, a security exception that expires before renewal, a cheaper tier with a missing required feature, and a business owner request unsupported by usage.

Participants may use AI for extraction, comparison, calculation, challenge, and drafting. They must anchor material claims, verify usage and price calculations, identify conflicts, compare renew, renegotiate, downgrade, replace, and hold options where relevant, and state the authorized next action.

Two calibrated reviewers score the seven dimensions independently. They double-score all high-ambiguity cases and every prohibited-condition flag. One case produces a disagreement: the recommendation to renegotiate is defensible, but one reviewer treats the missing feature as a verification failure while the other treats it as an explicitly escalated uncertainty. Adjudication shows that the rubric does not distinguish verified absence from unresolved availability. The guide is revised prospectively, and the original ratings remain in the record.

The report shows dimension distributions, four prohibited-condition flags, source-reconstruction results, disagreement, missing AI traces in two cases, and review effort. It states that the evidence supports interpretation for the defined synthetic renewal cases under the tested configuration. It does not claim predictive validity, live savings, organization-wide decision capability, or transfer to regulated systems.

Failure conditions

Hold the method when the population is undefined, cases do not represent important exceptions, evidence keys are created after seeing outputs, reviewers score with later outcomes visible, anchors reward style over evidence, failed cases are omitted, missing records become low scores, or agreement is reported without examining what reviewers agreed about.

Reject interpretations that equate prompt count with orchestration, citation count with evidence quality, speed with decision quality, reviewer agreement with correctness, or later outcome with what was knowable at the cutoff.

Two Prune boundary

Two Prune can run realistic work simulations and preserve evidence about observed participant performance across context framing, evidence navigation, AI orchestration, verification and risk control, judgment and synthesis, and communication. It does not provide a universal decision-quality measure, establish causal business impact, certify controls, or make employment decisions. Customers retain responsibility for measurement design, decision authority, tools, data, interpretation, and use of assessment evidence.

Sources

  1. 1.NIST AI RMF Core · National Institute of Standards and Technology
  2. 2.HM Treasury: Magenta Book 2026 · UK Government
  3. 3.GAO AI Accountability Framework · U.S. Government Accountability Office