Measuring AI verification skill in knowledge work

Define observable verification behavior

Start with a realistic deliverable and seed it with a bounded mix of conditions: a correct claim, an unsupported claim, a stale source, a conflicting source, a calculation error, an ambiguous instruction, and an item that requires escalation. Do not require every trap in every exercise. Choose conditions that fit the task and give the participant a fair opportunity to demonstrate the behavior.

Define scoring anchors before observing work. Distinguish noticing a risk, selecting an appropriate check, performing it correctly, interpreting the result, correcting the output, and recording uncertainty. A participant should not receive full credit merely for writing “verify this” or adding a source that does not support the claim.

The U.S. Government Accountability Office's AI Accountability Framework organizes practices around governance, data, performance, and monitoring and includes questions and procedures for assessors. Its system-accountability focus is broader than worker skill, but it supports examining evidence, performance, and ongoing monitoring rather than accepting an output at face value.

Sources: [1]

Construct the verification trace

Create one row for each material decision unit in the deliverable. Record the original claim or calculation, consequence if wrong, source cited by the AI, authoritative reference used for checking, action performed, discrepancy found, correction made, remaining uncertainty, reviewer note, and final disposition. Capture enough workflow evidence to understand the action without assuming that every unobserved check did not happen.

Match the method to the failure. Recalculate quantitative work from inputs; trace quotations to the original passage; compare dates and versions; inspect the primary record; test a formula with a known case; seek a second independent source when the first is unclear; and escalate when authority or evidence is insufficient. Counting browser visits or prompts does not reveal whether these checks were suitable or correctly interpreted.

The NIST AI Risk Management Framework Core describes governance, mapping, measurement, and management activities, including documented testing, independent review, feedback, and monitoring. It offers useful risk-management prompts but does not provide a validated individual skill score for a particular assessment.

Sources: [2]

Worked artifact: a synthetic budget recommendation

Imagine a synthetic analyst asked to recommend whether a programme budget should be renewed. An assistant drafts a memo claiming a 22% cost reduction and cites a spreadsheet summary. The verification trace marks the percentage as material because it drives the recommendation. The participant opens the workbook, rebuilds the denominator, and finds that the assistant compared a nine-month period with a full year.

The participant corrects the comparison to matched periods, adds a note that contractor costs are missing, and changes the recommendation from “renew at the current level” to “hold the decision until the missing invoices are reconciled.” A second claim about satisfaction is traced to a survey with a low response rate; it remains descriptive and is not presented as representative of the full group.

The trace gives credit for locating the decision-driving claims, selecting suitable checks, correcting the calculation, and communicating unresolved evidence. It does not infer skill from the polished memo alone. The example is synthetic and is not a customer result or a mature benchmark.

Review the measure

Use multiple scenarios with different failure patterns so that a participant cannot succeed by repeating one ritual. Check whether instructions, access, time, and source availability create comparable opportunities. Have reviewers apply the anchors to the same sample, discuss disagreements, and revise ambiguous anchors before interpreting scores broadly.

Report observed behavior, evidence strength, and missingness separately. Two Prune can evaluate task-bounded verification evidence from controlled simulations for human interpretation. It does not prove future workplace performance, establish that AI use will be safe, or make automated employment decisions.