AI capability measurement guide for the AI academy owner

The owner is an orchestrator, not the sole authority

The AI academy owner usually coordinates a programme across learning, business, technology, risk, privacy, security, and workforce stakeholders. Ownership of the academy does not automatically confer authority over legal interpretation, production access, tool approval, or employment decisions.

NIST's AI Risk Management Framework says processes for operator and practitioner proficiency should be defined, assessed, and documented. It also calls for clear roles, human oversight, ongoing monitoring, and risk management tied to the context in which an AI system is used. The EU AI Act's current Article 4 similarly says AI-literacy measures should account for people's technical knowledge, experience, education, training, and the context of use. The European Commission's practice repository is useful for examples, but it explicitly says copying those practices does not create a presumption of compliance.

This guide is educational and not legal advice. A qualified owner should confirm current jurisdiction-specific obligations.

Use the SCOPE decision agenda

The SCOPE agenda turns five programme decisions into evidence requests.

### Situation

Define the work context before selecting content.

Output: a dated context map, not a generic list of AI topics.

### Capability

Describe observable performance at the point of work.

Output: a capability map with evidence statements such as, "can identify an unsupported claim in a generated supplier brief and route it before external use."

### Options

Compare programme mechanisms against the evidence needed.

| Option | Useful for | Weak evidence if used alone | | --- | --- | --- | | Self-paced modules | Shared vocabulary and baseline concepts | Attendance does not show performance in work | | Tool demonstrations | Feature orientation | Following a demo does not establish transfer | | Workshops | Guided practice and peer discussion | Facilitator support may mask independent gaps | | Work simulations | Observable decisions, verification, and delivery | One scenario does not prove universal capability | | Manager observation | Contextual follow-through | Ratings may be inconsistent without a common rubric |

Output: a delivery mix with a stated role for each mechanism.

### Proof

Specify evidence before launching the programme.

Output: an evidence plan that distinguishes participation, demonstrated performance, workplace application, and business outcomes. Those constructs should not be collapsed into one adoption score.

### Escalation

Name the decisions the academy owner cannot make alone.

Output: an authority map with owner, reviewer, collaborator, and informed roles.

• Which approved AI systems and workflows are in scope?
• What decisions, data, customers, or operations could be affected?
• Which groups use, supervise, acquire, or govern the systems?
• What errors or misuse would matter most in each context?
• What should a person be able to frame, investigate, create, verify, and communicate?
• Which actions require domain expertise or approval?
• What does safe escalation look like?
• Which skills are shared, and which are workflow-specific?
• What artifact or behaviour will show the capability?
• Which rubric and evidence rules will reviewers use?
• What baseline or comparison is appropriate?
• How will missing evidence and uncertainty be reported?
• What programme change follows each result?
• Legal or regulatory interpretation goes to qualified counsel.
• Tool and data access goes to the relevant technology, security, privacy, and business owners.
• Production controls remain with accountable operational owners.
• Assessment evidence should not become an automated employment decision.
• Claims about business impact require a design that can address confounds and alternative explanations.

Worked artifact: a six-week pilot decision map

Consider an original synthetic academy for 24 procurement and operations professionals who use an approved assistant to compare supplier documents and draft internal recommendations.

| Decision | Evidence requested | Owner | Pilot rule | | --- | --- | --- | --- | | Who participates? | Workflow exposure, role need, approved access | Business owner | Include two common workflows; do not select people by assumed aptitude | | What is taught? | Source comparison, provenance, calculation checks, escalation | Academy owner with domain and risk reviewers | Exclude unapproved production automations | | How is practice observed? | Two realistic simulations with source packs and planted contradictions | Assessment owner | Score work process and deliverable evidence | | What counts as progress? | Better detection and correction against the same rubric, with uncertainty shown | Measurement owner | Report small-sample results descriptively; no mature benchmark claim | | What changes next? | Error patterns, feedback, missing controls, transfer observations | Steering group | Revise scenarios or support before expanding scope |

The first simulation asks participants to compare supplier proposals containing a unit mismatch, an outdated policy excerpt, and an AI-generated recommendation with an invented certification. The second uses a different source pack but measures the same underlying capabilities. Reviewers retain the source map, prompts, revisions, calculation checks, escalation choices, and final recommendation.

The pilot does not claim that six weeks prevents incidents or improves enterprise performance. It asks a narrower question: can participants demonstrate the specified behaviours in representative tasks, and what evidence is missing before programme expansion?

Every role, participant count, workflow, and result in this example is synthetic.

Reporting boundaries

Report at four separate levels:

Do not label completion as capability, capability as adoption, or adoption as business impact. Do not present a small internal pilot as a validated benchmark. Preserve missingness, reviewer disagreement, scenario limits, and changes to tools or policy.

• Reach: who received access and completed programme activities.
• Demonstration: what participants did in bounded tasks and how strong the evidence was.
• Transfer: whether managers or process evidence show relevant behaviours in approved work.
• Outcome: operational effects evaluated with an appropriate design and explicit confounds.

Questions for the steering meeting

In a Two Prune assessment, realistic source packs, workspace activity, revisions, and final deliverables can provide task-bounded evidence across the six AI fluency pillars. Two Prune does not certify regulatory compliance, promise risk reduction, rank people, predict workplace success, or make employment decisions.

• Which work decisions justify the academy now?
• Which errors must participants learn to detect or escalate?
• What evidence is unavailable from existing learning analytics?
• Which stakeholders own the tools, data, controls, and affected workflows?
• What would cause the pilot to stop, narrow, or change?
• Which claims are we explicitly not prepared to make?

Sources

NIST: AI Risk Management Framework Core

• NIST AI RMF Playbook: Map
• EUR-Lex: Regulation (EU) 2024/1689, Article 4
• European Commission: AI talent, skills and literacy

Sources: [1] [2] [3] [4]