AI Assessment Design Templates: A Practical Starter Pack

Use a six-part design brief

Start with six fields: intended use, target capability, task, evidence opportunity, output, and boundary. Intended use states the decision the assessment informs. Target capability names what is being observed. The task creates a realistic opportunity to show it. Evidence opportunity specifies what participants can do or produce. Output defines the work product. Boundary says what the result cannot establish.

NIST’s AI RMF Core recommends documenting intended purpose, context, human oversight, metrics, and limits. It also distinguishes testing and evaluation from unsupported claims of general validity (AI RMF Core). The UK Government Service Manual advises using relevant, believable tasks with clear success conditions in usability benchmarking (benchmarking guidance). These sources inform the design discipline, not validation of a particular assessment.

Sources: [1] [2]

Template 1: task and evidence map

Use one row per observable behavior:

| Field | Prompt | | --- | --- | | Capability | What bounded behavior matters? | | Opportunity | Where can the participant demonstrate it? | | Evidence | What action or artifact could show it? | | Counterevidence | What would weaken the interpretation? | | Missingness | What if the opportunity or capture is absent? | | Reviewer note | What judgment must remain human? |

The map prevents a rubric from scoring behavior the task never allowed. If source verification matters, include sources with checkable claims and enough time to inspect them.

Template 2: scenario integrity card

Record the scenario version, audience, briefing, inputs, expected deliverable, time limit, permitted tools, accessibility adjustments, designed ambiguity, evidence traps, and prohibited uses. Add a source register for every supplied fact. Ask AI to identify contradictions, accidental clues, inaccessible formats, and paths that allow completion without demonstrating the target behavior.

Worked example

Consider a synthetic 30-minute supplier-response review. The intended use is curriculum improvement after training. The task includes a requirement table, an outdated policy excerpt, and an email that conflicts with the table. The deliverable is a correction memo with source links.

The evidence map gives participants opportunities to frame scope, reconcile evidence, use AI, verify claims, exercise judgment, and communicate corrections. One criterion examines whether the participant resolves the conflict or marks it uncertain. If telemetry fails, the report marks process evidence missing rather than scoring the participant down.

A failed template says, “Score critical thinking from 1 to 5.” It neither defines the construct nor points to observable evidence. Repair it by naming the decision, behavior, opportunity, artifact, and anchors.

Template 3: review and interpretation record

For each criterion, write observable anchors, evidence sources, disqualifying ambiguity, and a confidence note. Include a reviewer calibration example and an escalation route for disagreement. Separate final-output evidence from workflow evidence. Do not infer an unobserved action.

Before use, verify:

These templates create a reviewable design record. They do not establish reliability, fairness, predictive validity, or benchmark maturity without separate evidence.

Retain rejected scenario versions so later reviewers can see which ambiguity, shortcut, or unsupported inference was deliberately removed and why.

• The task matches the intended decision and work context.
• Every criterion has a fair opportunity and observable evidence.
• Sources, calculations, and contradictions are reproducible.
• Missing evidence lowers confidence, not participant performance.
• Accessibility and data-handling requirements are tested.
• Interpretation remains task-bounded and human-reviewed.
• No result becomes an automated employment decision.