Practical guide
How to report AI capability evidence to an assessment researcher
Give the researcher a claim-evidence record: intended interpretation, observed episode, scoring rationale, uncertainty, missingness, review history, comparability limits, and the decision boundary for sound decisions.
Give the researcher a claim-evidence record: intended interpretation, observed episode, scoring rationale, uncertainty, missingness, review history, comparability limits, and the decision boundary for sound decisions.
Report to the research mandate
Classify the assessment researcher's role for this report. They may own the measurement argument, influence task and rubric design, review score evidence, or collaborate with domain, accessibility, privacy, security, and programme owners. They are not automatically the budget owner or final authority for customer use.
The report should let the researcher answer four questions:
The Standards for Educational and Psychological Testing state that reports should support valid interpretation and use, describe what the assessment covers, explain what scores represent and their precision or reliability, and use language appropriate to the audience. They also caution against reporting individual scores without appropriate evidence for the intended interpretation. Standards: reporting and interpretation
- What interpretation is being made?
- Which observed evidence supports it?
- How stable and complete is that evidence?
- Which uses remain outside the evidence?
Sources: [1]
Use a CLAIM report structure
### C — Context and intended claim
Start with assessment title, scenario, version, administration date, time limit, tools, allowed support, accommodation, expected deliverable, and benchmark family if one is actually available. Then write the narrow interpretation.
Prefer: “In this assessment, the participant compared two policy sources and removed an unsupported statement before submission.”
Avoid: “The participant is reliable with AI.” The second statement generalizes across tasks, tools, settings, and time without evidence.
### L — Lineage from evidence to score
For every material score statement, retain:
Do not paste an event log and call it explanation. The lineage should show why the event matters to the criterion and how the final output agrees or conflicts with observed process.
### A — Agreement, uncertainty, and missingness
Report reviewer agreement by criterion and explain the resolution path for material disagreements. Separate score precision, reviewer agreement, evidence completeness, and benchmark uncertainty; one cannot substitute for another.
Missing evidence must remain visible. A failed capture event, unsupported file type, abandoned attempt, or inaccessible material is not automatically poor participant performance. Record whether missingness is operational, participant-driven, condition-specific, or unresolved.
The Standards note that score reports should communicate uncertainty and relevant contextual information, and that empirical research with report users can improve clarity. Use comprehension testing for the actual audience rather than assuming a technically accurate report will be interpreted correctly. Standards for Educational and Psychological Testing
### I — Interpretation and comparison limits
State what each comparison population contains, its period, inclusion rules, size, scenario family, and relevant administration conditions. Label percentiles directional or unavailable when the reference group is early, sparse, or mismatched.
Keep overall and contextual comparisons separate. A contextual percentile may be more relevant to the scenario but less stable. Never invent a benchmark because the report layout expects one.
List excluded interpretations explicitly, such as productivity, future job performance, broad personality traits, causal training effects, or an employment recommendation.
### M — Modifications, governance, and next evidence
Close with assessment version, rubric version, model or scoring configuration where relevant, reviewers, adjudication state, report generator version, corrections, and release approval. Name the next evidence needed to reduce uncertainty: a second task, manual artifact review, accommodation review, reviewer recalibration, or a follow-up conversation.
This change history matters because score meaning can shift when scenarios, evidence capture, rubric definitions, reference groups, or generated explanations change.
- event or artifact identifier;
- source opened or data used;
- relevant prompt, note, calculation, revision, or deliverable excerpt;
- rubric criterion and level;
- reviewer rationale;
- counterevidence or alternative explanation;
- confidence or evidence-strength note.
Sources: [2]
Worked artifact: claim-evidence reporting table
A synthetic customer-operations assessment asks a participant to review an inbox, two policy documents, account data, and an AI-assisted response. The researcher receives this bounded table:
| Report claim | Supporting evidence | Counterevidence or missingness | Review state | Boundary | | --- | --- | --- | --- | --- | | The participant detected an outdated refund rule in this simulation | Opened both dated policies; note cites the current section; draft changed from 30 to 14 days | Source-open telemetry for the older file arrived 18 seconds late | Two reviewers agree; telemetry issue logged | Does not establish source checking across other tasks | | The participant verified the account calculation before delivery | Spreadsheet formula, recalculated total, note explaining the discrepancy, corrected final amount | No external calculator event captured | Reviewers agree on artifact evidence; process route partly missing | Supports this calculation episode only | | Communication score reflects a clear customer-facing explanation | Final response states the decision, policy basis, next action, and uncertainty | First draft was overly certain but was revised | One-level initial disagreement resolved against anchor example | Not a measure of general writing ability |
The report then adds:
The example is synthetic. The number is not a validated benchmark, evidence about a real person, or proof that Two Prune predicts workplace outcomes.
- overall assessment score: 72, with the scoring scale and rubric version;
- evidence strength: moderate, because the final artifacts are complete but one telemetry event is delayed;
- contextual percentile: unavailable, because the relevant completed reference group is too small;
- calibration note: reviewers agreed after consulting the approved anchor; disagreement history retained;
- follow-up question: ask the participant how they would verify a policy source when document dates conflict and telemetry is unavailable.
Audit the report before release
The researcher should be able to reproduce each material statement from retained evidence. Check that:
The Society for Industrial and Organizational Psychology's position statement points users back to professional assessment standards and warns about discrimination risks in AI-enabled assessment contexts. That supports careful review and context; it does not turn one report into proof of fairness or validity. SIOP position statement
Two Prune reports observed assessment performance and evidence behind the score. It does not issue employment recommendations, replace human judgment, or establish general capability from one assessment. AI may help organize evidence and draft explanations, but accountable reviewers own interpretations, uncertainty, corrections, and release.
- intended interpretations and uses are named;
- each score statement has inspectable support;
- contrary evidence and missingness are visible;
- reviewer disagreement and adjudication are retained;
- score confidence is not confused with participant confidence;
- comparison populations and dates are disclosed;
- generated summaries do not exceed the underlying evidence;
- accessibility and accommodation conditions are recorded appropriately;
- report, rubric, task, and generator versions are linked;
- prohibited downstream interpretations are explicit.
Sources: [3]
Sources
AERA, APA and NCME: Standards for Educational and Psychological Testing
- Standards excerpt: Test administration, scoring, reporting, and interpretation
- Standards for Educational and Psychological Testing: official site
- SIOP: position statement referencing professional assessment standards
Sources: [2] [1] [4] [3]
Sources
- 1.Standards: reporting and interpretation · www.testingstandards.net
- 2.Standards for Educational and Psychological Testing · www.testingstandards.net
- 3.SIOP position statement · www.siop.org
- 4.Standards for Educational and Psychological Testing: official site · www.testingstandards.net