Measuring the quality of AI-enabled knowledge work

Define quality from the user's decision

Start with the deliverable, intended user, decision it supports, required sources, constraints, and consequence if it is wrong. Translate “good quality” into observable criteria such as factual support, calculation correctness, coverage of mandatory requirements, treatment of uncertainty, usability, traceability, and appropriate escalation.

Weight criteria by consequence rather than appearance. A minor formatting defect and a fabricated financial figure should not cancel each other in an average. Define non-compensable failures that require correction or a hold, and distinguish them from preferences that can be recorded without blocking use.

The U.S. Government Accountability Office's AI Accountability Framework organizes practices around governance, data, performance, and monitoring. Its questions and audit procedures support examining objectives, evidence, performance, and oversight, although it is not an individual-work scoring standard.

Sources: [1]

Construct the quality acceptance map

Create one row for each material requirement. Include the user need, criterion, evidence source, AI contribution, human contribution, test, severity, acceptance anchor, observed result, uncertainty, correction owner, and final disposition. Preserve the source version and the delivered artifact so another reviewer can reproduce the judgment.

Review both the output and the available process evidence. The deliverable shows what was produced; the process can show whether sources were opened, calculations rebuilt, conflicts noticed, or limitations added. Missing process evidence is a confidence limitation, not proof that a worker failed to perform a check elsewhere.

Define the acceptance context as well. An exploratory note, an external statement, and an operational instruction may require different evidence and review even when they discuss the same subject. Record the permitted use on the artifact so a limited draft is not silently reused for a higher-consequence decision.

The NIST AI Risk Management Framework Core describes mapping context, measuring risks and impacts, documenting testing, seeking independent review, and monitoring performance. These practices support a structured quality review but do not establish a universal benchmark for all knowledge work.

Sources: [2]

Worked artifact: a synthetic market-entry memo

Imagine a synthetic strategy analyst using an assistant to draft a market-entry memo. The quality acceptance map contains seven material requirements: current market definition, traceable source dates, correct currency conversion, competitor coverage, explicit assumptions, risk treatment, and a recommendation tied to evidence.

The memo is fluent and complete-looking, but the map finds two release-blocking defects. A market-size figure comes from an undated secondary summary, and the currency calculation mixes monthly and annual revenue. The analyst traces the figure to a current primary publication, rebuilds the calculation from labeled inputs, and narrows a claim where geographic coverage remains incomplete.

The revised memo passes the factual and calculation anchors but retains a limitation: one competitor's private pricing cannot be verified. The reviewer accepts the memo for an exploratory discussion, not for a final investment decision. The result records which criteria passed, what changed, and what remains unresolved. This example is synthetic and is not evidence of customer impact or a mature quality benchmark.

Verify the measure itself

Use varied deliverables and failure patterns, then ask multiple reviewers to apply the prewritten anchors to the same sample. Investigate disagreements, criterion ambiguity, missing evidence, and opportunities that the task did not provide. Revisit criteria when the audience, source environment, tool, workflow, or consequence changes.

Report quality dimensions, blocking defects, confidence, and limitations separately instead of hiding them in one number. Two Prune can provide task-bounded process-and-output evidence from controlled simulations for human interpretation. It does not prove general workplace performance or make automated employment decisions.