Practical guide
AI capability pilot design for an assessment researcher
Design the pilot as an evidence study: set claims, sample real work, calibrate review, inspect missingness, test report use, and gate each next decision explicitly.
Design the pilot as an evidence study: set claims, sample real work, calibrate review, inspect missingness, test report use, and gate each next decision explicitly.
Give the researcher a defined mandate
Classify the assessment researcher as an owner, influencer, reviewer, or collaborator for each pilot decision. They may own the measurement argument and study protocol, influence task and rubric design, review score evidence, and collaborate with domain, accessibility, privacy, security, and programme owners. That does not make them the budget owner or authorize the eventual customer use.
The pilot should answer a measurement question, not stage a sales demonstration. Begin with the exact interpretation a reader may make, the use of that interpretation, and the evidence needed to support or reject it.
The Standards for Educational and Psychological Testing organize professional responsibilities around validity evidence for intended interpretations and uses, reliability or precision, fairness, administration, scoring, reporting, and consequences. These concerns make predeclared interpretations and limits central to the pilot. Standards for Educational and Psychological Testing
Sources: [1]
Use a PILOT protocol
### P — Predeclare interpretations and decisions
For each intended use, record:
Example: “In this simulation, the participant identified two unsupported claims and corrected both before submission” is testable. “The participant is safe with AI” is not supported by one simulation.
### I — Instantiate authentic work and administration cells
Build tasks from representative inputs, constraints, tools, ambiguities, and deliverables. Document the relationship between the simulation and target work rather than relying on a role label.
Create an administration matrix before collecting data. Cells may vary only when the pilot is designed to study that variation:
| Cell | Tool access | Time | Materials | Review condition | | --- | --- | --- | --- | --- | | A | Standard workspace and approved AI support | 45 minutes | Fixed case v1 | Two independent reviewers | | B | Same as A | 45 minutes | Fixed case v1 | Adjudication sample only | | Accessibility | Approved accommodation | Documented | Equivalent case | Same rubric plus access review |
Record device, interruptions, support events, incomplete attempts, and version changes. Do not mix materially different conditions into one result without analysis.
### L — Label evidence and calibrate reviewers
Define each rubric level with observable evidence and counterexamples. Give reviewers anchor submissions that illustrate both clear and borderline cases. Then run calibration before live review:
The Standards' official site identifies the joint AERA, APA, and NCME publication as professional guidance for testing. Use the full standards and qualified review for the intended context; the source is not a shortcut to declaring a pilot validated. Official Standards site
### O — Observe operation, missingness, and interpretation
The pilot must test more than whether people can finish the task. Track:
Missingness is evidence about the pilot. Do not silently convert absent process evidence into poor capability, or remove incomplete attempts until the pattern looks cleaner.
### T — Test gates and transfer limits
At the decision review, compare results with the predeclared gates:
Advancing means the evidence supports a next step, not that the assessment has proven productivity, training impact, fairness across all groups, mature benchmarks, or financial return.
- the narrow statement a report reader might make;
- the task and process evidence expected to support it;
- plausible alternative explanations;
- the decision the pilot may inform;
- the decisions explicitly outside scope;
- advance, revise, and stop criteria.
- reviewers score the same anchor set independently;
- they record evidence references and confidence, not only numbers;
- disagreements are classified as rubric ambiguity, missing evidence, reviewer error, or legitimate judgment;
- the team revises training or rubric language, then repeats the exercise;
- unresolved disagreement rules are documented before live scoring.
- task completion, timing, and support events;
- whether required workspace evidence is actually captured;
- rubric coverage and reviewer disagreements;
- missing evidence by task, condition, and participant group;
- report statements that readers overgeneralize or misunderstand;
- access, privacy, security, and operational incidents;
- cases where the final output and observed process point in different directions.
- advance: administration is reproducible, required evidence is reconstructable, review can be calibrated, report language is understood within bounds, and no material unresolved issue blocks the intended next study;
- revise: a specific task, capture, rubric, reviewer, access, or reporting defect can be corrected and retested;
- stop: the intended interpretation cannot be supported, essential evidence is unavailable, administration is not defensible, or the proposed use exceeds the assessment's boundary.
Sources: [2]
Worked protocol: policy-response simulation
A synthetic customer-operations programme wants to learn whether a structured assessment can capture evidence of source verification. The researcher creates one simulation in which participants review a customer email, two policy documents with different dates, a small account table, and an AI-assisted draft response.
The protocol defines three observations:
| Intended observation | Required evidence | Failure condition | Decision gate | | --- | --- | --- | --- | | Participant notices the outdated policy | Opened sources, note or prompt, revision trail, final response | The platform cannot distinguish which policy informed the answer | Revise evidence capture | | Reviewer can justify the verification score | Rubric anchors, cited episode, confidence note, independent second review | Material disagreements cannot be traced to evidence or rubric wording | Stop score use until repaired | | Report reader understands the limit | Comprehension interview using the report | Reader converts a task-bounded result into a general employment recommendation | Rewrite report and retest |
The pilot uses 24 synthetic submissions spanning clear success, clear failure, incomplete evidence, conflicting final-output and process signals, and approved accommodations. The number is chosen to exercise workflow paths and reviewer calibration; it is not claimed to validate the assessment or establish a benchmark. All files are synthetic, and no participant outcome is used for an employment decision.
The researcher maintains an issue ledger with condition, affected records, severity, owner, correction, retest evidence, and disposition. Four missing source-open events trigger a capture investigation. Three reviewer disagreements trace to an ambiguous rubric phrase. Two report readers overgeneralize a percentile placeholder, so the placeholder is removed. The result is “revise and rerun,” not “pilot succeeded” because completion was high.
Report what the pilot actually established
The final memo should list attempted, completed, excluded, and reviewed records; administration conditions; task versions; evidence-capture results; reviewer calibration; missingness; incidents; comprehension findings; advance, revise, and stop dispositions; and limitations. Preserve negative results and decisions that remain unmade.
Two Prune provides realistic AI-enabled work simulations and evidence-backed assessment reporting. It does not make employment decisions, prove general workplace performance from one assessment, or convert a bounded operational pilot into validation evidence by assertion.
Sources
AERA, APA and NCME: Standards for Educational and Psychological Testing
- Standards for Educational and Psychological Testing: official site
- SIOP: position statement referencing professional assessment standards
Sources: [1] [2] [3]
Sources
- 1.Standards for Educational and Psychological Testing · www.testingstandards.net
- 2.Official Standards site · www.testingstandards.net
- 3.SIOP: position statement referencing professional assessment standards · www.siop.org