Measuring AI Judgment With Decision Forks, Not Confidence Scores

Define the judgment behavior

Start with a specific behavior, such as distinguishing fact from assumption, choosing when to verify, escalating a material uncertainty, or balancing speed against evidence quality. Describe the task conditions that create a fair opportunity to show it. Avoid broad labels such as “good judgment” that allow reviewers to reward style, confidence, or familiarity instead of the intended behavior.

NIST’s AI RMF describes mapping context, measuring identified risks, and documenting knowledge limits as parts of risk management (NIST AI RMF). It supports observing context-sensitive choices. It does not provide a validated score for individual AI fluency or authorize employment decisions.

Sources: [1]

Design three decision forks

Create a realistic task with three moments where reasonable options diverge. At each fork, record the available evidence, time remaining, possible actions, consequence of error, and what additional information could reduce uncertainty. Score only observable indicators tied to the construct: source checking, qualification, contradiction handling, escalation, and revision after new evidence.

The CDC Program Evaluation Framework describes evaluation as systematic collection and analysis and begins with assessing context before describing the programme or focusing questions (CDC Program Evaluation Framework, 2024). That logic supports documenting the context and intended use of the measure before interpreting results.

Sources: [2]

Worked artifact: supplier recommendation case

Imagine a synthetic operations case with two suppliers. One has the lower headline price; the other has better delivery evidence. Midway through the task, a late email reveals that one price excludes a required service. The first decision fork tests whether the participant defines the decision criteria before asking AI for a recommendation. The second tests whether they reconcile the new evidence. The third tests whether the final recommendation distinguishes verified facts, assumptions, and follow-up conditions.

The scoring record cites the participant’s actions and deliverable. A reviewer notes that the participant found the exclusion, revised the comparison, and conditioned the recommendation on a missing capacity check. The report states what was observed in this task and where evidence was absent. It does not infer a stable personality trait or future workplace performance.

Check interpretation before reporting

Have a second reviewer apply the indicators to selected records, investigate disagreements, and refine ambiguous anchors. Examine whether every participant received the same opportunity to encounter each fork. Track missing evidence separately from weak performance. If the task, assistance, or scoring instructions change, treat comparability as uncertain until checked.

Avoid rewarding confidence and verbosity

A participant can make a sound bounded decision in a short note or an unsound one in polished prose. Anchors should point to evidence use, revision, qualification, and action, not tone. Reviewers should not infer an unseen check from a correct answer or infer poor judgment merely because the available evidence was incomplete.

Use a disagreement log during calibration. For each disputed indicator, record the cited event, competing interpretations, final rule, and whether earlier records need reconsideration. This makes scoring changes inspectable and prevents an informal preference from silently becoming part of the construct.

This is an educational measurement design, not a validated benchmark. It supports task-bounded evidence for human interpretation and follow-up questions. It must not produce automated hiring recommendations, rank people across unlike tasks, or claim that one simulation proves performance outside the assessment.

When reporting development areas, point to the observed fork and a practice opportunity. Avoid personality labels. A useful follow-up might ask the participant to explain what evidence would change the recommendation or to repeat a calculation with a new assumption. The objective is clearer evidence, not a disguised pass threshold.