Questions to Ask Before Designing an AI Work Assessment

Questions about purpose and boundaries

Who will use the result? What decision or discussion will it support? Is the purpose learning, programme evaluation, development, research, or another approved use? What must the result never be used to decide? What evidence would make the result too uncertain to interpret?

NIST’s AI RMF Core recommends documenting intended purpose, context, human oversight, metrics, and limitations (AI RMF Core). The UK Government Service Manual recommends realistic tasks with clear success conditions for benchmarking (usability benchmarking). These sources support design discipline, not validation of a specific assessment.

Sources: [1] [2]

Questions about the task opportunity

What authentic work decision anchors the scenario? Which inputs, ambiguities, constraints, and tradeoffs make it representative? Does the task create a fair opportunity to frame context, inspect evidence, use AI, verify output, exercise judgment, and communicate? Could someone complete it through a shortcut that bypasses the target behavior?

Ask whether time, reading load, tool access, language, and interface demands introduce irrelevant difficulty. Define permitted resources and accessibility adjustments. A behavior cannot be scored when the task never gives a reasonable opportunity to show it.

Questions about evidence and scoring

What actions and artifacts can support each criterion? Which evidence comes from the deliverable, captured workflow, or reviewer observation? What evidence would contradict a favorable interpretation? How will missing telemetry be handled? Are anchors observable, or do they rely on vague labels such as “excellent judgment”?

Separate no evidence from negative evidence. If a participant never encounters a conflict, failure to resolve one is not poor performance. If the capture system fails, confidence should fall rather than the score.

Worked question-to-design example

Imagine a synthetic 25-minute operations assessment. The design question is whether a post-training exercise can reveal how participants reconcile contradictory incident records. The source pack includes a dashboard extract, two emails, and an outdated procedure. The output is an escalation note with an evidence table.

The design questions expose a flaw: the current rubric scores calculation checking, but the scenario contains no calculation. The team either adds a relevant calculation with a reproducible answer or removes that criterion. Another question reveals that the outdated procedure is visually marked “old,” making the evidence trap trivial. The team revises the artifact while preserving accessibility.

A weak process begins with a generic rubric and swaps role labels across scenarios. The repaired process starts with the decision, construct, task opportunity, and evidence.

Questions about review and use

Who has the expertise and authority to approve the scenario? How will reviewers calibrate on examples? How are disagreements recorded? Which changes trigger re-review? How will participants understand data use, time rules, and available support? What follow-up question should a result generate rather than answer automatically?

Before launch, confirm:

These questions improve design transparency. They do not establish reliability, fairness, benchmark maturity, or prediction of performance beyond the assessment.

Keep a versioned design log linking each revised task, source, rubric anchor, accessibility finding, and reviewer decision. A future update should show why the scenario changed and which evidence requires fresh review.

• Intended use and prohibited use are explicit.
• Every criterion has an authentic opportunity and evidence source.
• Sources, calculations, and expected facts are verified.
• Missingness and confidence rules are written.
• Accessibility and privacy are tested with the actual interface.
• Reviewers can cite evidence for each judgment.
• Results stay task-bounded and human-interpreted.