Can AI capability be measured with a quiz?

Decide what claim the score is supposed to support

Start with a one-sentence claim. “The learner can identify which data class must not be entered into an unapproved tool” is a plausible quiz claim. “The learner uses AI safely and effectively at work” is too broad for a short set of selected-response questions.

Evidence-centered assessment design asks three linked questions: what knowledge or capability matters, what observable evidence would support the claim, and what task can elicit that evidence. The ETS overview of evidence-centered design used for the TOEFL iBT explains that assessment design requires explicit measurement claims and close examination of the evidence supporting them. The lesson transfers to AI capability measurement: do not begin with a convenient question format and invent the capability claim afterward.

Write the intended use, population, conditions, stakes, and limitations beside the claim. A low-stakes learning check, a programme diagnostic, and an individual capability decision need different evidence. If the score will affect a consequential decision, qualified measurement, legal, accessibility, and domain review may be necessary.

Sources: [1]

Use the MATCH test for quiz-suitable claims

The MATCH test helps decide whether a quiz is an appropriate instrument.

A quiz fits when meaning is narrow, action distance is small, coverage is deliberate, conditions are relevant, and interpretation is bounded. It becomes weak evidence when a person can select the expected answer without producing or verifying work.

The PISA 2025 Technical Standards state that tasks should elicit evidence of capacity in the assessed domain, use knowledge intrinsic to that domain, fit the question being asked, and collectively cover the area under investigation. PISA is not an AI-fluency test and its standards do not validate a workplace assessment. They provide a useful primary example of aligning tasks, evidence, construct coverage, and administration.

• Meaning: Is the construct defined narrowly enough to test?
• Action distance: How far is choosing an answer from performing the real action?
• Task coverage: Do the items cover the important parts of the construct rather than one memorable topic?
• Conditions: Do item conditions resemble the context in which the claim will be used?
• Human interpretation: Are score meaning, uncertainty, and next actions explicit?

Sources: [2]

Separate knowledge evidence from performance evidence

Create an evidence map before choosing item types. Put each claim into one of three columns.

Knowledge evidence includes recognizing a policy boundary, identifying a hallucination risk, selecting an authoritative source, or interpreting a confidence statement. A well-designed quiz can often elicit this evidence.

Judgment evidence includes weighing conflicting sources, choosing when to escalate, or explaining a tradeoff. Scenario-based written responses may provide some evidence, but the scoring rubric and range of situations matter.

Performance evidence includes researching, prompting, revising, checking calculations, managing files, and delivering an artifact under realistic constraints. This generally requires a work sample, simulation, observation, or portfolio evidence rather than a quiz alone.

The categories can overlap. The point is not that quizzes are bad. The point is that the evidence format must match the claim.

Worked example: the same policy tested two ways

Suppose an AI programme wants to assess whether participants can handle unsupported claims in an executive briefing. A four-option quiz asks what to do when an AI-generated market statistic has no source. The participant selects “find an authoritative source or remove and qualify the claim.” That response supports a bounded claim: the participant recognized the expected verification rule in the described situation.

Now give the participant a synthetic briefing task. The evidence folder contains an old market report, a current regulator release, a spreadsheet with a denominator error, and an AI draft containing an invented statistic. The participant has 25 minutes to produce a one-page briefing using an approved AI assistant. The assessor can observe which sources were opened, how the model was instructed, whether the calculation was checked, what happened to the invented statistic, and how residual uncertainty was communicated.

The participant may answer the quiz correctly and still retain the invented statistic under time pressure. Or the participant may miss the vocabulary in the quiz but perform the verification reliably in the task. The two instruments answer different questions. The quiz is a knowledge check; the simulation creates an opportunity to observe applied behavior in a bounded context.

The evidence artifact is a claim-to-task matrix with four fields for each claim: expected evidence, task condition, scoring rule, and limitation. For the quiz item, the limitation says “recognition in a described case, not observed workplace execution.” For the simulation, it says “performance in one synthetic task, not proof of future behavior across settings.”

Combine instruments without manufacturing a single truth score

Use quiz results to locate concept gaps, select follow-up tasks, or compare learning before and after instruction when the design supports that use. Use performance tasks to inspect process and output together. Use interviews or reflection to probe reasoning that the activity log cannot reveal. Keep the measures separate until there is evidence that combining them produces a meaningful score.

Report item coverage, administration conditions, scoring method, missing data, accessibility accommodations, and uncertainty. A high quiz percentage is not automatically a proficiency level, benchmark percentile, or capability certification. A change after training is not automatically caused by the training. Reused items, coaching, guessing, language demands, and tool familiarity can affect results.

Choose the smallest instrument that fits the decision

For a five-minute policy refresher, a quiz may be enough. For a development conversation, pair it with one scenario response. For a claim about applied AI-enabled work, use a realistic task that elicits evidence navigation, AI orchestration, verification, judgment, and communication. For high-stakes use, require stronger technical evidence, independent review, fairness analysis, and explicit human interpretation.

Two Prune uses timed, realistic simulations to create task-bounded evidence across six AI-fluency pillars. It can complement a quiz by showing what a participant did with supplied evidence and AI support. It does not prove general job performance, provide mature norms without data, rank people automatically, or make employment decisions.