Practical guide
AI capability measurement guide for the assessment researcher
An assessment researcher protects the connection between a work simulation, the evidence it produces, and the interpretation reported to a customer. The role is usually an owner or reviewer of measurement quality. It is not automatically the budget owner, programme sponsor, or employment decision-maker. This agenda turns that mandate into recurring decisions. It is intended for a researcher.
An assessment researcher protects the connection between a work simulation, the evidence it produces, and the interpretation reported to a customer. The role is usually an owner or reviewer of measurement quality. It is not automatically the budget owner, programme sponsor, or employment decision-maker. This agenda turns that mandate into recurring decisions. It is intended for a researcher reviewing an assessment programme, not for claiming that Two Prune or any other instrument has already demonstrated validity, reliability, fairness, or business outcomes.
Establish the mandate before reviewing metrics
Record who owns the assessment purpose, case design, scoring rules, technical operation, privacy review, accessibility, customer reporting, and change approval. State which decisions the researcher can make, which they can recommend, and which require another owner.
Write the intended interpretation in plain language. For example: “Observed performance in this synthetic operations task provides evidence about how a participant frames a constrained problem, navigates supplied evidence, works with AI support, verifies outputs, and communicates a deliverable.” Do not expand that statement into a prediction of performance in every role or an automated employment decision.
NIST AI 800-3 treats measurement as a discipline of explicit targets, assumptions, statistical reasoning, uncertainty, and limitations. NIST's AI Risk Management Framework similarly distinguishes validity and reliability and asks whether test sets and conditions are representative of the context of use. Those principles support the agenda below, but they do not validate a local assessment by citation alone.
Run the seven-decision agenda
### 1. Interpretation and use
Required evidence: approved purpose statement, score interpretation, prohibited uses, stakeholder map, and reporting language.
Stop condition: stakeholders cannot agree on what a score means or begin treating it as a universal trait.
### 2. Task and evidence opportunity
Required evidence: task specification, content map, expected evidence chain, expert review, pilot observations, and version history.
Stop condition: an important score dimension has no observable task opportunity or depends on information not available to the participant.
### 3. Scoring and evidence lineage
Required evidence: scoring rubric, evidence schema, scoring examples, reviewer guidance, model or rule version, and adjudication records.
Stop condition: a score changes without an attributable evidence or version change.
### 4. Consistency and uncertainty
Required evidence: double-scored samples, disagreement analysis, repeatability checks where appropriate, uncertainty analysis, and calibration notes.
Stop condition: disagreement clusters around a core criterion and the reporting language still implies precise measurement.
### 5. Validity and unintended strategies
NIST's August 2026 TEVV-Athlon Framework is an initial public draft. Its structure of objectives, events, tools, and modular evaluation blocks is a useful design reference, not a settled standard. NIST's public work on evaluation cheating also shows why implementation loopholes and transcript review matter when an apparent score may reflect the evaluation setup rather than the target capability.
Required evidence: validity argument, challenge cases, transcript or activity review, relationship studies where justified, and documented rival explanations.
Stop condition: an unintended strategy succeeds often enough to threaten the interpretation.
### 6. Fairness, accessibility, privacy, and failure impact
Required evidence: accessibility review, privacy basis and retention rules, subgroup diagnostics with denominators, incident records, and recovery procedure.
Stop condition: a barrier or system failure materially affects interpretation and cannot be separated from participant performance.
### 7. Reporting and change control
Required evidence: report specification, benchmark definitions, release checklist, change-impact assessment, monitoring plan, and issue log.
Stop condition: a material change ships under an old measurement claim without review.
- What precise capability, behaviour, or work product is being interpreted?
- Which decisions may use the result, and which are out of scope?
- What alternative explanations could produce the same score?
- What confidence or calibration note must accompany the report?
- Does the simulation give a participant a real opportunity to demonstrate each claimed capability?
- Are briefing, documents, inbox, research, AI support, time constraints, and deliverable aligned to the intended work?
- Do irrelevant difficulty, reading load, tool friction, or hidden knowledge dominate the result?
- Are evidence traps and ambiguity realistic rather than arbitrary?
- Can every score trace to rubric criteria and observed evidence?
- Are process signals distinguished from final-deliverable quality?
- How are missing, contradictory, or low-quality signals handled?
- Are automated analyses reviewable and bounded?
- Does the rubric reward useful judgment rather than superficial keyword presence?
- Do reviewers apply criteria consistently to the same evidence?
- Are repeated or parallel tasks comparable enough for their stated use?
- Which sources of variation come from participant performance, task form, reviewer, system, or missing data?
- Is the sample sufficient for the precision implied by the report?
- Does the evidence support the intended interpretation rather than a convenient proxy?
- Do scores relate to expert judgments or other relevant evidence in the expected way?
- Can participants obtain high scores through shortcuts, memorised phrasing, excessive length, or tool exploits?
- Do changes in tools or case materials alter what is being measured?
- Do task content, language, tools, timing, and accommodations introduce barriers unrelated to the intended construct?
- Are subgroup analyses appropriate for the sample and reported with uncertainty?
- Are data collection and retention proportionate to the assessment purpose?
- What happens when the system fails, evidence is missing, or a participant needs support?
- Does the report explain what was measured, evidence strength, uncertainty, benchmark limits, and missing evidence?
- Are overall and contextual benchmarks clearly distinguished?
- Are early percentiles labelled directional or unavailable?
- Which task, rubric, scoring, model, or interface changes require re-evaluation?
- Who approves release and who monitors post-release incidents?
Worked artifact: quarterly decision meeting
A synthetic operations assessment enters its quarterly review with four changes: one source spreadsheet was replaced, an AI tool outage affected 11 of 84 sessions, reviewers disagree on the Verification and Risk Control pillar in 7 of 20 double-scored records, and accessibility status is missing for 23 participants.
| Agenda item | Evidence | Researcher decision | Owner and next gate | | --- | --- | --- | --- | | Task version | Content map and change diff | Treat the spreadsheet replacement as a new task version until comparability is tested | Case owner runs expert review and a bridge sample | | Tool outage | Event logs and affected sessions | Do not interpret affected process signals as participant performance | Operations owner repairs records; reporting adds a limitation | | Reviewer disagreement | Double scores and adjudication notes | Pause precise pillar comparisons; clarify two rubric anchors and recalibrate reviewers | Measurement owner repeats double scoring before release | | Missing accessibility data | Data dictionary and missingness table | Do not claim a subgroup comparison; investigate collection and consent path | Privacy and accessibility owners decide a proportionate repair | | Benchmark | Completion counts by version | Keep contextual percentile directional and separate versions | Researcher reviews after bridge evidence is available |
The meeting does not “average away” the four problems. It preserves usable evidence, limits affected interpretations, assigns repairs, and sets an evidence gate for renewed reporting. The sponsor decides whether the programme timeline changes; the researcher owns or reviews the measurement recommendation within the agreed mandate.
Pilot and reporting boundaries
Before broader use, pilot the complete workflow, including instructions, tools, evidence capture, scoring, review, reporting, incidents, and accommodations. Predefine the questions the pilot can answer. Small samples can reveal workflow defects and scoring disagreements, but they do not support precise population claims merely because a percentage can be calculated.
Report the assessment version, intended interpretation, evidence available, scoring method, uncertainty, incidents, exclusions, benchmark population and date, known limitations, and next review. Separate verified findings from hypotheses and proposed studies.
Two Prune can provide assessment evidence about observed performance in realistic AI-enabled work simulations. It does not make employment decisions, replace human judgment, or turn an assessment research agenda into proof of predictive or business outcomes.
Sources
NIST AI 800-3, A Measurement Guide for Information Technology Systems
- NIST AI Risk Management Framework, Characteristics of Trustworthy AI Systems
- NIST, TEVV-Athlon Framework for Evaluating AI Systems
- NIST CAISI, Background: AI models can cheat on evaluations
Sources: [1] [2] [3] [4]
Sources
- 1.NIST AI 800-3, A Measurement Guide for Information Technology Systems · National Institute of Standards and Technology
- 2.NIST AI Risk Management Framework, Characteristics of Trustworthy AI Systems · National Institute of Standards and Technology
- 3.NIST, TEVV-Athlon Framework for Evaluating AI Systems · National Institute of Standards and Technology
- 4.NIST CAISI, Background: AI models can cheat on evaluations · National Institute of Standards and Technology