Measuring AI Literacy With an Evidence Ladder

Define the learning target in observable terms

Replace “understands AI” with behaviors such as identifying a suitable use, explaining a limitation, protecting restricted data, checking a source, testing a calculation, or revising an output after contradictory evidence. State the task, assistance, time, and evidence available. Each behavior needs a fair opportunity to appear before it can be interpreted.

UNESCO’s AI Competency Framework for Students describes progression across understand, apply, and create and includes human-centred mindset, ethics, techniques, and system design (UNESCO student AI competency framework). It offers a useful progression model, though an enterprise programme must adapt it to adult work and actual roles.

Sources: [1]

Build four evidence levels

Level one is explanation: the learner can describe a concept and boundary. Level two is choice: they select an appropriate method in a scenario. Level three is performance: they complete a realistic task and leave inspectable evidence. Level four is transfer: they apply the behavior in a different task or approved work context. Report the highest observed level for each target, with missingness and confidence.

UNESCO’s AI Competency Framework for Teachers covers knowledge, skills, values, human agency, ethics, applications, pedagogy, and professional development (UNESCO teacher AI competency framework). Its emphasis on professional learning supports treating literacy as developmental rather than a permanent badge.

Sources: [2]

Worked artifact: source-verification ladder

Imagine a synthetic programme target: “verify an AI-assisted factual summary.” At level one, the learner explains why a fluent answer can be unsupported. At level two, they choose source read-across over asking the same model to check itself. At level three, they correct a summary containing a superseded policy date and cite the current source. At level four, they apply the method to a different document set several weeks later.

The evidence record includes the scenario, participant actions, final artifact, source checks, reviewer notes, and unobserved behaviors. A correct final answer without an observable check may support delivery quality but not the verification indicator. The programme does not infer a stable trait from one task.

Review consistency and access

Calibrate reviewers on concrete examples and log disagreements. Confirm that learners received comparable essential instructions and tool access. Provide appropriate accessibility support and record material differences. If scenarios or scoring anchors change, treat trend comparisons cautiously until the relationship is checked.

Use results for development

Tie each gap to a practice opportunity and a follow-up question. Aggregate patterns can inform curriculum design, while individual records remain task-bounded. Keep attendance, self-report, telemetry, performance, and workplace outcomes as separate evidence classes.

Prevent the ladder from becoming a badge

Set a review condition for each evidence level. Tool behavior, policies, and work tasks change, so a past performance record should retain its date and context. Do not display the highest level as a permanent personal label. When a new task requires different evidence or consequences, design a new opportunity to demonstrate the behavior.

Audit whether the progression rewards access to prior experience rather than learning. If one group routinely enters with more familiar tools, interpret programme change separately from absolute performance. The assessment can inform targeted support, but it should not hide unequal opportunity behind a single literacy score.

Where a learner uses a method that differs from the expected path, score the observable outcome against the stated behavior rather than enforcing one preferred prompt sequence. Preserve enough workflow evidence for the reviewer to distinguish a sound alternative from an unsupported lucky answer.

This educational measurement design does not create a mature benchmark, certify literacy, or authorize automated employment decisions. Interpretations require the programme’s construct, evidence, reviewers, and current operating context.