Measuring AI proficiency in knowledge work

Define proficiency at the task level

Specify what capable performance requires. For a research task, it may involve framing the question, selecting authoritative sources, directing AI to synthesize bounded evidence, checking citations, resolving conflicts, and writing a useful recommendation. For a spreadsheet task, it may involve inspecting formulas, asking AI for explanations, independently reperformance of material calculations, and communicating limitations.

Separate constructs that can look similar. Tool familiarity is not evidence navigation. Long prompts are not necessarily orchestration. A correct final answer may conceal an unsafe process, while a participant who catches and repairs an AI error may demonstrate stronger verification.

The OECD's Artificial Intelligence and the Future of Skills programme develops bounded ways to compare AI capabilities with occupational requirements. Its policy research is not an individual workplace scoring rubric, but it reinforces the need to define capabilities and contexts precisely.

Sources: [1]

Build a proficiency evidence matrix

Create one row per capability. Record the scenario feature that creates an opportunity, the observable behavior, the associated deliverable evidence, and anchored descriptions of weak, developing, and strong task-bounded performance. Add rules for partial, contradictory, and missing evidence.

For verification, the scenario might contain a plausible unsupported statistic. Strong evidence would require locating the problem, checking the underlying source, changing the conclusion, and preserving the correction. Merely writing “verify sources” in a plan is weaker evidence. If no material source conflict exists, the task cannot strongly assess conflict resolution.

The NIST AI RMF Measure guidance recommends selecting methods and metrics appropriate to mapped risks and documenting what cannot be measured. It also treats AI performance as sociotechnical and context-dependent. Use that principle to report evidence limits rather than fill gaps with assumptions.

Sources: [2]

Worked artifact: a synthetic procurement analysis

Imagine a synthetic participant comparing three software proposals from a requirements sheet, pricing table, security note, and stakeholder email. The proficiency matrix creates opportunities for task framing, evidence navigation, AI orchestration, verification, judgment, and communication.

The participant asks AI to rank vendors, then notices that one quoted price excludes implementation. They recalculate total cost, trace a claimed certification to the security note, and discover that the evidence covers a different product tier. They revise the recommendation, flag the unresolved security requirement, and present a conditional decision instead of selecting a definitive winner.

The matrix links each score observation to the opened sources, calculation, AI interaction, revision, and final memo. It also records that no accessibility issue appeared in the scenario, so accessibility judgment is not assessed. This is a synthetic example, not a benchmark or evidence of performance outside the task.

Calibrate scoring and confidence

Have multiple reviewers apply the anchors to a varied sample and compare disagreements. Check whether evidence supports the same interpretation across scenarios and participants. Examine whether domain knowledge, task difficulty, language, accessibility, or tool familiarity changes the opportunity to demonstrate the intended capability.

Report an overall interpretation only when component evidence is sufficient. Distinguish absence of evidence from observed weak behavior. Where telemetry is incomplete or a scenario lacks an opportunity, lower confidence or mark the construct unavailable rather than assigning a confident low score.

Verify the measurement process

Trace every rating to observable evidence and attempt to reproduce calculations. Check versioned scenario materials, instructions, tools, rubric, and reviewer guidance. Revisit anchors after changes to task design, AI features, allowed assistance, or available telemetry. Monitor for drift and unexpected shortcuts that alter what the task measures.

This approach supports interpretable, task-bounded proficiency evidence. It does not establish a universal trait, mature population norm, or prediction of future work performance. Two Prune produces structured assessment evidence with confidence limitations for human interpretation and does not make employment decisions.