Assessment confidence in AI-enabled knowledge work

Define the inference before measuring confidence

State the claim the assessment is intended to support. For example: during a timed analytical simulation, the participant identified conflicting sources, corrected a calculation, and narrowed a recommendation. Do not silently expand that observation into a claim about every work setting, long-term performance, or an employment decision.

Map each scored capability to an opportunity in the task. If a scenario never requires escalation, absence of escalation evidence is missing opportunity rather than weak performance. Record whether evidence comes from the final deliverable, observed workflow, source use, AI interaction, revision, calculation, or reviewer note.

NIST defines reliability in measurement as repeatability and consistency. Reliability matters, but consistent scoring does not by itself establish that the assessment measures the intended capability. Confidence needs separate evidence about task coverage, evidence capture, scoring agreement, and interpretation boundaries.

Sources: [1]

Assemble a confidence dossier

For each reported result, record the intended construct, observable behavior, opportunity to demonstrate it, evidence captured, missing evidence, scoring anchor, reviewer agreement, task conditions, potential alternative explanation, and permitted interpretation. Keep confidence components separate rather than blending them into one unexplained value.

A useful dossier can report:

The NIST AI RMF Measure function calls for documented test sets, methods, uncertainty, performance in conditions similar to deployment, independent review, and limits on generalization. It is a voluntary AI risk framework rather than a psychometric standard, but it supports disciplined reporting of measurement context and uncertainty.

• coverage: which capabilities the task actually elicited;
• evidence completeness: whether the relevant process and output were captured;
• scoring stability: whether the same evidence receives consistent judgments;
• anchor fit: whether rubric descriptions distinguish materially different performance;
• context fit: how closely the simulation resembles the interpretation being made; and
• calibration status: whether the comparison group and sample support a percentile or benchmark.

Sources: [2]

Worked artifact: a synthetic operations assessment

In a synthetic assessment, a participant receives a service-volume spreadsheet, two conflicting manager notes, and a request for a staffing recommendation. They detect duplicate weeks, ask AI to recalculate, compare the result with the source rows, and deliver a bounded recommendation. They never face a privacy-sensitive input or a need to escalate outside the task.

The confidence dossier records strong evidence for source reconciliation and calculation checking, moderate evidence for judgment because only one tradeoff was presented, and no evidence for privacy handling or formal escalation. A second reviewer independently applies the scoring anchors; one disagreement concerns whether the participant verified the corrected total or merely accepted the AI output. The reviewers inspect the event and calculation evidence, resolve the discrepancy, and retain the rationale.

The report does not average “no opportunity” into a lower score. It states the unobserved areas and suggests a different follow-up task if those capabilities matter. The example is synthetic and does not establish validated reliability, predictive validity, or benchmark precision.

Test and communicate confidence

Use repeated scoring on a stratified sample, including boundary cases and incomplete evidence. Examine agreement by pillar and anchor, not only overall score. When reviewers disagree, classify whether the cause is an ambiguous rubric, missing evidence, interface presentation, task design, or genuine judgment difference. Revise the appropriate layer and test again.

Check for construct drift when tools or workplace practices change. A task that once required manual prompt composition may no longer elicit the same orchestration behavior when an assistant automatically plans steps. Reconfirm that task opportunities and scoring anchors still represent the claimed capability.

Show confidence limitations beside the result: evidence gaps, small or mismatched benchmark groups, single-task sampling, changed task conditions, and unresolved reviewer disagreement. Two Prune provides evidence-backed assessment signals from bounded simulations. It does not make employment decisions, promise future performance, or claim mature percentiles where supporting data is insufficient.