Assessment Researchers and AI-Enabled Work: An Evidence-Argument Method

Start with the intended interpretation

Write one bounded sentence describing what a result may support. For example: “Within this timed simulation, the record describes how the participant framed the task, used available evidence, verified material claims, and communicated a deliverable.” Avoid jumping from one exercise to general workplace performance. Name the population, task family, conditions, intended use, and prohibited use.

Assessment strategy guidance from the U.S. Office of Personnel Management emphasizes beginning with the competencies to be assessed and considering reliability, validity evidence, process design, resources, and trained raters (OPM assessment strategy). Two Prune is not an employment decision system, so the relevant lesson is disciplined linkage between the construct, task, and evidence rather than OPM’s selection context.

Sources: [1]

Build a five-link evidence argument

Use five linked records. The construct record defines the behavior of interest. The opportunity record shows where the task gives a participant a fair chance to demonstrate it. The observation record lists what can actually be captured. The interpretation record explains how observations support a score or narrative. The limitation record identifies missingness, alternative explanations, and decisions the evidence cannot support.

For each link, write a plausible rebuttal. A source-opening event might reflect evidence navigation, but it might also be accidental. A polished deliverable may reflect communication skill, but it may conceal weak verification. A silent event log may mean a behavior did not occur, or that the system did not capture it. The scoring design should respond by combining evidence and lowering confidence when critical opportunities or observations are absent.

Worked artifact: a verification-evidence matrix

Consider a synthetic policy-analysis simulation. One row of the matrix concerns “checks an externally attributable claim.” The opportunity is a brief containing a plausible but outdated statistic. Observable evidence includes opening the source, replacing the statistic, adding a citation, or documenting uncertainty. The scoring anchor distinguishes copying the claim, noticing but not resolving it, resolving it with a current source, and explicitly limiting the conclusion.

The rebuttal column notes that a corrected final sentence does not prove the participant performed the check unaided. The confidence rule therefore considers available workflow evidence and the final artifact together. If source-view events are unavailable, the report can describe the corrected output while withholding a strong process inference. The follow-up question asks how the participant decided the original source was insufficient.

Review reliability without pretending it is fixed

Create paired sample cases that differ on one material feature, then ask reviewers to score independently and cite the evidence behind each judgment. Compare disagreements at the evidence and anchor level. Revise ambiguous wording, clarify missingness rules, and repeat the exercise. Record who participated, which version they used, and what changed.

NIST’s AI Risk Management Framework Core organizes AI risk work around govern, map, measure, and manage, with context and documented limits central to measurement (NIST AI RMF Core). That supports a continuous review process. A small calibration exercise is process evidence, not proof that scores are stable across populations, tasks, reviewers, or time.

Sources: [2]

Questions the researcher should keep open

Ask whether the task creates comparable opportunities; whether irrelevant reading or language demands distort the intended construct; whether captured signals are complete enough for the interpretation; whether reviewers can identify the same evidence; whether subgroup patterns require investigation; and whether users understand confidence limits. Treat these as design and research questions rather than automatic pass conditions.

The assessment researcher should document evidence, disagreements, and next studies while preserving human interpretation. The role does not certify the product, set an employment threshold, or claim that task-bounded observations predict future performance.