Measuring AI Fluency Through Comparable Work Evidence

Define fluency as behavior in context

Separate six questions: Did the participant frame the assignment? Find and weigh relevant evidence? Direct and refine AI use? Verify material outputs? Exercise judgment across tradeoffs? Deliver something useful to the stated audience? Each question needs observable anchors within the task rather than personality adjectives or assumptions about intent.

UNESCO’s AI Competency Framework for Teachers describes competencies across a human-centred mindset, ethics, foundations and applications, pedagogy, and professional learning, with progression levels rather than one undifferentiated label (UNESCO AI competency framework). Although written for educators, it illustrates why context, values, application, and development should remain visible instead of collapsing into tool familiarity.

Sources: [1]

Build an opportunity map before a score

For each behavior, identify the exact moment the exercise permits it. A verification item might place a plausible unsupported figure in a brief. An evidence-navigation item might include two sources that disagree. A judgment item might require choosing between speed and reversibility. Record the expected evidence, plausible alternative explanations, and what missing capture means.

Do not score absence when the opportunity did not occur. If the relevant contradiction was accidentally removed from a case, the participant had no comparable chance to demonstrate resolution. Mark the evidence unavailable and lower confidence. This distinction prevents scenario defects from becoming participant findings.

Worked artifact: an AI-fluency evidence card

Consider a synthetic thirty-minute operations exercise. The participant must recommend whether to repair or replace a failing intake process. Available materials include a cost table, service log, security note, and an outdated vendor claim.

The evidence card records the task version and six rows. Context framing cites the participant’s stated decision and constraints. Evidence navigation cites the documents opened and conflicting dates noticed. AI orchestration describes how the participant decomposed comparison and drafting work. Verification cites a recalculation and replacement of the outdated claim. Judgment records the reversible repair recommendation and the unresolved migration assumption. Communication cites the final decision memo and action owner.

Each row includes observed evidence, counterevidence, opportunity quality, missingness, anchor, reviewer rationale, and confidence. The card does not convert a single missing event into failure. It does not claim that the result applies to unrelated work.

Calibrate the interpretation

Ask multiple reviewers to score paired synthetic records that vary on one behavior. Require evidence citations. Inspect where disagreement comes from: ambiguous anchors, different assumptions, missing data, or genuine judgment. Revise the task or rubric and preserve the change log. Revisit interpretation when task content, AI tools, capture systems, or intended uses change.

NIST’s AI RMF Core organizes work into govern, map, measure, and manage and treats measurement as contextual, documented, and connected to human oversight (NIST AI RMF Core). This supports monitoring the assessment system itself, not merely producing participant scores.

Sources: [2]

Report what is known and unknown

Show the task, conditions, observed behaviors, supporting evidence, contrary evidence, confidence limitations, and follow-up questions. If benchmarks are immature, label comparisons directional or unavailable. Separate performance within the simulation from any hypothesis about workplace transfer. Avoid automated employment recommendations or universal readiness labels.

Control instrumentation drift

Maintain a measure-change register for event capture, task materials, tool access, scoring anchors, and reviewer instructions. Each entry should identify the old and new version, reason, affected records, comparison impact, and owner. When a desktop update begins capturing a source-view event that was previously missing, do not report the resulting increase as changed behavior. Mark the break and restrict comparisons to periods with sufficiently comparable observation. Run a known synthetic case after each material instrumentation change and verify which events appear, which remain unavailable, and how the report describes missing evidence.

The result is a structured account of observed work. Establishing reliability, validity, fairness, benchmark stability, or predictive use requires additional studies with relevant tasks, populations, reviewers, and outcomes; it cannot be inferred from the existence of a rubric.