Practical guide
How an assessment researcher can build the case for applied AI capability evidence
An assessment researcher links each intended claim to observed evidence, a bounded pilot, clear limits, named owners, and decision gates agreed before any work begins.
An assessment researcher links each intended claim to observed evidence, a bounded pilot, clear limits, named owners, and decision gates agreed before any work begins.
Classify the researcher's role
Use four labels before preparing the case:
An assessment researcher may hold several of these roles. They are not automatically the budget owner, implementation owner, or final decision-maker. Naming the role prevents a technical review from being mistaken for programme approval.
The case is not “AI is important, so buy an AI assessment.” It asks whether a defined pilot can produce usable evidence for a specified interpretation in a specified context.
The Standards for Educational and Psychological Testing state that score interpretations for intended uses require validity evidence and that test users should consider reliability or precision, applicable norms, and possible consequences. That makes the intended interpretation the starting point for the case, rather than a feature list or a generic claim that the assessment is “scientific.” Standards for Educational and Psychological Testing
- owner: accountable for the measurement argument, study plan, analysis, and limitations;
- influencer: shapes scenario realism, rubric definitions, evidence capture, and reporting language;
- reviewer: challenges validity, reliability, fairness, administration, and interpretation;
- collaborator: works with programme, domain, privacy, accessibility, security, and operational owners.
Sources: [1]
Build a USE-EVIDENCE decision map
Create one row for every decision-facing interpretation:
| Field | Question | | --- | --- | | Use | What will the customer do with the assessment evidence? | | Statement | What exact interpretation might a reader make? | | Evidence | Which task, process signal, artifact, score, or review supports it? | | Evaluation | How will the pilot test coverage, scoring, consistency, and alternative explanations? | | Need | What operational value would usable evidence provide? | | Caveat | Which populations, contexts, outcomes, or decisions are outside the evidence? | | Owner | Who can approve the use, investigate the evidence, and act on a limitation? |
Separate observed performance from broad capability. A participant may demonstrate source checking in one timed simulation. That does not establish how they perform across every task, tool, team, or future setting. Separate score precision from benchmark comparisons too. A consistent scoring process does not create a mature reference population.
Compare the real alternatives
The relevant alternatives are usually not “buy or do nothing.” They may include self-report, manager observation, training completion, tool-usage analytics, work samples, structured simulations, or a combined design. Evaluate each against the required interpretation:
Manager observation may provide contextual evidence but can be sparse, inconsistent, and confounded by opportunity. Tool analytics can show events but not necessarily purpose or quality. Self-report can capture perceptions but not observed task performance. A simulation can provide standardized evidence, while still remaining a sample of behaviour under specified conditions.
- Can the method expose the work process as well as the final output?
- Can it distinguish source use, AI orchestration, verification, judgment, and communication?
- Is the task representative enough for the intended context?
- Can a reviewer inspect the evidence behind a score?
- Which groups, accommodations, tools, or conditions may change performance?
- What administration effort, review expertise, and governance does it require?
Worked decision map: customer-operations pilot
Consider a synthetic customer-operations team evaluating an assessment in which participants review an inbox, policy documents, a spreadsheet, and AI-assisted draft replies. The researcher proposes three pilot interpretations:
| Use | Statement | Evidence | Pilot evaluation | Caveat | | --- | --- | --- | --- | --- | | Development planning | Participant identified and corrected unsupported policy claims in the simulation | Source-opening trail, claim notes, revisions, final reply, rubric evidence | Double-review 30 submissions; inspect disagreements and missing evidence | Task-bounded observation, not a general risk-control trait | | Programme diagnosis | The cohort showed recurring difficulty connecting AI output to authoritative policy sources | Aggregated item and evidence patterns | Predefine aggregation rules; inspect task effects and missingness | No causal claim about training or tool quality | | Follow-up conversation | The report identifies a specific verification episode worth discussing | Evidence excerpt, score rationale, confidence note | Review whether the excerpt supports the wording and whether alternate explanations are visible | Not an automated employment recommendation |
The pilot uses 30 synthetic submissions only to test administration, evidence capture, reviewer agreement, missingness, and report comprehension. The number is not presented as a sufficient validation sample or benchmark. The researcher records how submissions were selected, which conditions were held constant, who reviewed them, and which interpretations remain unsupported.
The Standards also emphasize that those who administer and score assessments need appropriate preparation. The case therefore budgets for reviewer training, calibration examples, disagreement resolution, and evidence audits rather than treating the scoring interface as self-validating.
Answer the predictable objections
“We already track tool adoption.” Usage may help describe exposure or frequency. It does not by itself show whether someone framed a task, selected evidence, checked an output, or produced a useful deliverable.
“Managers can observe this.” Manager evidence may be valuable, but opportunity, task assignment, visibility, and rating practice can vary. The pilot should test whether a structured work sample adds evidence that ordinary observation does not reliably capture.
“One overall score is easier.” Simplicity is useful only if the score's meaning and uncertainty remain clear. Preserve pillar evidence, task context, confidence limitations, and traceable examples rather than implying more precision than the data support.
“A pilot should prove return on investment.” A measurement pilot can establish whether the task runs, evidence is captured, scoring can be reviewed, and reports are interpreted as intended. It cannot by itself establish organizational productivity, training impact, or financial return.
Use evidence gates, not purchase theatre
Define advance, revise, and stop conditions before collecting results:
The business case should request a bounded pilot, named owners, necessary review time, and a decision date. It should not promise proven outcomes. Two Prune is an assessment product for realistic AI-enabled work simulations and evidence-backed scoring. It does not make employment decisions, replace human judgment, or establish general workplace performance from one assessment.
- advance if task coverage is acceptable, evidence can be reconstructed, reviewers can apply the rubric with manageable disagreement, reports are understood within their boundaries, and no material unresolved access or fairness issue blocks the use;
- revise if a specific task, rubric, workflow, or report defect can be corrected and retested;
- stop if the intended interpretation cannot be supported, required evidence is unavailable, administration conditions are not defensible, or the proposed use exceeds the product boundary.
Sources
AERA, APA and NCME: Standards for Educational and Psychological Testing
- Standards for Educational and Psychological Testing: official site
- Society for Industrial and Organizational Psychology: position statement referencing current professional assessment standards
Sources: [1] [2] [3]
Sources
- 1.Standards for Educational and Psychological Testing · www.testingstandards.net
- 2.Standards for Educational and Psychological Testing: official site · www.testingstandards.net
- 3.Society for Industrial and Organizational Psychology: position statement referencing current professional assessment standards · www.siop.org