Practical guide
AI capability assessment blueprint: free template and guide
An AI capability assessment blueprint is the design record that connects a defined capability to realistic tasks, observable evidence, scoring rules, administration conditions, and interpretation limits. It prevents a team from starting with attractive questions or a generic AI quiz and deciding what the result means afterward. This copyable blueprint uses DESIGN: define the construct, establish.
An AI capability assessment blueprint is the design record that connects a defined capability to realistic tasks, observable evidence, scoring rules, administration conditions, and interpretation limits. It prevents a team from starting with attractive questions or a generic AI quiz and deciding what the result means afterward. This copyable blueprint uses DESIGN: define the construct, establish the use, specify tasks, identify evidence, govern scoring, and name limitations.
Source basis and boundary
IFAC's 2026 International Education Standard 6 treats assessment as the evaluation of professional competence and emphasizes assessment activities with sufficient reliability, validity, equity, transparency, and sufficiency. NIST's AI Risk Management Framework and Generative AI Profile support documenting context, risks, measurement methods, human oversight, and limitations. The UK Government's AQuA Book supports proportionate assurance, verification, traceable assumptions, and independent review. These sources inform the design controls below. They do not validate a particular assessment or establish that its scores predict performance outside the assessed task. IFAC IES 6 NIST AI RMF AQuA Book
Sources: [1] [2] [3]
Copyable DESIGN blueprint
### Blueprint control
| Field | Entry | | --- | --- | | Assessment title, owner, version, status | | | Intended purpose and decision context | | | Participant population and prerequisites | | | Administration mode and time limit | | | Scenario and evidence-set owner | | | Scoring owner, reviewers, approver | | | Review date and change trigger | |
### D - Define the capability
Write a construct statement that names the capability, its components, and its exclusions.
This assessment creates an opportunity to observe [capability] while a participant completes [bounded task] using [available evidence and tools]. It does not establish [material non-claim].
| Construct component | Observable behaviour | Excluded interpretation | | --- | --- | --- | | | | |
Avoid defining capability as tool frequency, confidence, prompt length, or final polish. Separate knowledge, process, judgment, verification, and communication where those distinctions matter.
### E - Establish the intended use
Name who will interpret the result, what question it may inform, and which uses are prohibited. Specify whether the assessment supports development, programme evaluation, calibration, or another bounded purpose. Record consequences of error and the level of evidence needed before use.
| Intended use | User | Required evidence | Prohibited inference | | --- | --- | --- | --- | | | | | |
### S - Specify authentic tasks and conditions
| Task | Work product | Inputs and tools | Ambiguity or tradeoff | Time | Construct coverage | | --- | --- | --- | --- | --- | --- | | | | | | | |
Tasks should require the target behaviour rather than reward topic familiarity alone. Define instructions, evidence access, AI permissions, accessibility accommodations, interruptions, submission rules, and what the participant is told about scoring. Include enough structure for comparable opportunities while preserving realistic judgment.
### I - Identify evidence and failure modes
Map each scoreable claim to observable evidence.
| Evidence claim | Direct evidence | Supporting evidence | Missingness rule | Rival explanation | | --- | --- | --- | --- | --- | | | | | | |
Evidence may include the submitted work, source use, calculations, revisions, AI interactions, verification actions, and timing when legitimately available. Do not infer a behaviour from silence when the task did not create a fair opportunity to show it. Plan traps carefully: they should reveal verification and judgment, not reward obscure knowledge or inaccessible conventions.
List principal design failures: construct underrepresentation, irrelevant difficulty, source ambiguity, unequal tool access, scorer drift, prompt leakage, over-weighting the final artifact, missing evidence treated as poor performance, and claims that exceed the task.
### G - Govern scoring and review
| Criterion | Performance evidence | Anchored levels | Weight | Confidence rule | Escalation | | --- | --- | --- | --- | --- | --- | | | | | | | |
Define scoring anchors with concrete evidence, not adjectives alone. Require scorers to cite evidence and distinguish observed behaviour from interpretation. Specify double-scoring or adjudication for consequential or uncertain cases, scorer training, calibration examples, conflict handling, and what happens when evidence is missing.
Plan quality checks: scenario walkthrough, source verification, calculation reproduction, accessibility review, rubric-to-task coverage, scorer agreement, unusual-response review, and change control. A high agreement number does not repair a task that measures the wrong construct.
### N - Name interpretation limits and maintenance
State the population, task, time, evidence, and conditions to which the score applies. Define confidence limitations, benchmark maturity, expiry, correction, and requalification after changes. Record what evidence would be needed before making a broader claim.
Worked example: fictional supplier-risk simulation
A programme team wants to assess whether participants can use AI to support a supplier-risk recommendation. The first concept asks participants to summarize three documents and gives most points to the final memo.
DESIGN defines the construct more precisely: framing the decision, selecting relevant evidence, using AI within the evidence boundary, checking calculations and claims, weighing tradeoffs, and communicating a recommendation. The intended use is a post-training development discussion, not a prediction of workplace performance.
The revised task supplies a contract excerpt, incident log, cost table, policy note, and an email containing an unsupported claim. Participants have 45 minutes and may use an AI assistant. The evidence map requires a decision frame, source-linked risk claims, a reproduced cost comparison, treatment of the unsupported email, and an explicit limitation.
The rubric uses evidence anchors. A strong verification response opens the original cost table, reproduces the material calculation, and corrects or excludes the unsupported claim. A weak response repeats the claim without checking it. Missing AI activity does not automatically reduce the score; the scorer evaluates whether the participant used available tools and evidence effectively for the task.
A pilot walkthrough finds that the policy note uses an undefined acronym that creates irrelevant difficulty. The team repairs the source, reruns the walkthrough, double-scores representative submissions, and records disagreements. The report states that the result describes observed performance in this simulation and requires human interpretation.
Practitioner QA
Does every score claim map to a task opportunity and observable evidence?
- Can a participant understand the instructions and permitted AI use?
- Are source traps relevant to the construct rather than trivia?
- Can scorers apply anchors without inventing missing evidence?
- Are accessibility, privacy, and security requirements built into administration?
- Are score confidence and interpretation limits visible?
- Is every material change versioned and rechecked?
Two Prune boundary
Two Prune provides realistic AI-enabled work simulations and structured, evidence-backed assessment reports for human interpretation. It does not make employment decisions, establish that a blueprint is valid, establish fair outcomes, or turn a task-bounded score into proof of future performance.
Sources
IFAC: IES 6, Assessment of Professional Competence
- NIST: AI Risk Management Framework
- GOV.UK: The AQuA Book
Sources: [1] [2] [3]
Sources
- 1.IFAC IES 6 · education.ifac.org
- 2.NIST AI RMF · National Institute of Standards and Technology
- 3.AQuA Book · UK Government