Measuring review burden in AI-enabled knowledge work

Define the unit of review

Choose a concrete output such as a customer response, forecast note, policy summary, proposal section, or research brief. Break it into review units that reflect the actual decision: claims, calculations, source uses, requirements, data-handling choices, or proposed actions. Page count and word count are weak proxies when two outputs have very different evidence demands.

Name the reviewer role, authority, source access, expected checks, service level, and stopping rule. Separate active inspection time from waiting for clarification, source retrieval, escalation, or system recovery. Separate work created by the original task from work created by avoidable output defects.

NIST's AI RMF Appendix C on human-AI interaction emphasizes defined roles, context, varying human-AI configurations, and the complexity of presenting system information to people. It supports context-aware workload analysis but does not prescribe a universal productivity measure.

Sources: [1]

Build the review-load trace

For each unit, record the draft element, consequence, required evidence, evidence availability, expected check, issue found, action taken, active minutes, waiting minutes, handoff, rework cycle, final disposition, and confidence limit. Use dispositions such as accepted after evidence check, corrected, escalated, recreated, removed, unresolved, or not reviewable.

Distinguish necessary assurance from avoidable burden. Source verification may be required even for a sound draft. Reconstructing absent calculations, chasing nonexistent citations, resolving permission failures, or correcting inconsistent terminology may represent avoidable review work. Do not infer low burden from rapid approval when the reviewer lacked source access or authority.

The NIST AI RMF Measure Playbook recommends context-relevant testing, documentation, feedback, monitoring, and independent review practices. These ideas support recording conditions and limitations rather than treating one average time as proof of safe performance.

Sources: [2]

Synthetic trace: an executive forecast note

A synthetic finance team reviews an AI-assisted forecast note containing eight claims, two calculations, and one recommendation. The first reviewer spends six active minutes checking the prose but cannot open the source workbook. The trace marks the calculations and recommendation as not reviewable; the quick approval time is not counted as completed assurance.

A second reviewer receives access. One calculation is correct, one mixes monthly and quarterly periods, and a stated driver cannot be found in the workbook. Correcting the period takes four minutes. Tracing and removing the unsupported driver takes nine minutes plus a handoff to the analyst. The final trace separates necessary source inspection from fourteen minutes of defect-related rework and records one unresolved assumption.

When the team tests a revised workflow that attaches calculation lineage and source cells, the same task requires less reconstruction. The comparison remains specific to this synthetic task, reviewer setup, and evidence access. It does not establish a universal time saving or customer outcome.

Interpret without rewarding weak review

Report distributions of active review time, waiting, rework cycles, severe issues, unavailable evidence, escalation, and unresolved units. Stratify by task type and consequence. Pair time with coverage and outcome dispositions so faster but incomplete review does not look superior.

Have an independent analyst reproduce a sample of traces from event records and artifacts. Reopen the measure when output format, model, evidence access, automation, reviewer role, task complexity, risk tolerance, or release authority changes. Use qualitative review for rare high-consequence failures that averages can hide.

Two Prune can create a bounded opportunity to observe review and verification behavior and present evidence for human interpretation. It does not establish organization-wide productivity, prove risk reduction, set staffing levels, or make employment decisions.