An AI Assessment Score Is Only as Trustworthy as Its Evidence Chain
A useful AI assessment connects the claim it makes to a task that can elicit the right behaviour, observable evidence, conservative interpretation and visible limits.
Two Prune Research··4 min read
An assessment score is the end of an argument. To trust it, you should be able to inspect every step that came before it.
Imagine an assessment report that says a participant is strong at verification. The number looks precise. The chart looks finished. But the task contained no conflicting evidence, the participant never had to check a material claim and the activity record does not show what they reviewed.
The problem is not the design of the chart. The task never created enough evidence to support the claim.
Start with the claim, not the score
Two Prune begins with a bounded question: how did this participant demonstrate AI fluency in this assessment context? We do not treat a score as a personality trait, an employment recommendation or proof of performance across every role. The assessment should support a narrower claim with observable evidence.
That posture aligns with a basic measurement principle in the NIST AI Risk Management Framework: methods should suit the risks and context, uncertainty should be documented, and measurement should provide a traceable basis for decisions.[1] It also prevents a report from sounding more certain than its evidence.
The evidence chain has five links
Capability: define the behaviour of interest.
Task: create a realistic assignment that gives the behaviour a chance to appear.
Observation: capture the submitted deliverable and relevant actions taken in the workspace.
Interpretation: use a versioned AI evaluator to interpret bounded evidence, then verify the evidence links and calculate ratings through governed controls.
Report: explain the result, readable evidence references, evidence availability, missing-rating explanations, explicit limitations and score methodology.
A weak link changes what the final score can mean. If the task contains no meaningful evidence conflict, it offers a weak test of Evidence Navigation. If participants cannot inspect or revise an AI output, it reveals little about Verification & Risk Control. If the activity record is incomplete, the report must not invent a process or treat missing capture as poor performance. If evidence remains materially insufficient after bounded recovery, the overall score is withheld and a score-free limited-evidence report is produced.
The work product and the work process can disagree
The final deliverable shows whether the participant produced something accurate, useful and suited to the audience. Process evidence records parts of the observed sequence: which documents were opened, what AI interactions were captured, whether claims were checked and whether revision followed new evidence.
Put the framework to work
See how your workforce demonstrates AI fluency.
Two Prune uses realistic work simulations to produce structured assessment evidence—not automated employment decisions.
Two Prune is beginning a research programme—and inviting academic collaboration—to develop task-bounded AI-fluency assessment into a responsible measurement method.
Neither is sufficient alone. A strong memo may be paired with a fragile recorded process. A disordered sequence may still include a valuable correction after an error appears. Recorded sequence does not establish intent, causation or a stable trait. In one anonymised participant case, a coherent and strongly authored report still carried a material contradiction because the recorded work used AI to synthesise the evidence but did not establish verification of the final claims. Read the full verification case note.
Uncertainty belongs in the result
Two Prune uses a versioned AI evaluator to interpret bounded evidence. Separate validation checks that every rating links back to the evidence, calculates the six pillar ratings and overall score, and controls whether the report can be published automatically. Publication occurs only after checks of the required report format, PDF, stored file and other gates pass. If evidence remains materially insufficient after bounded recovery, the overall score is withheld and a score-free limited-evidence report is produced. Missing capture is never turned into poor performance or zero.
Population benchmarks are currently unavailable. Any future benchmark should remain unavailable or be explicitly directional until its comparison group is sufficiently mature. This is not a claim of completed psychometric validation. Calibration is an ongoing empirical programme: compare scoring across cases and reviewers, examine whether tasks elicit the intended behaviours, monitor evidence coverage and revise the evaluator contract or scoring rules when they produce unstable interpretations.
NIST describes AI evaluation as a continuing practice involving quantitative, qualitative or mixed methods, formal reporting and measures of uncertainty.[1] Its ARIA programme also focuses on testing AI systems in realistic societal contexts rather than treating model performance alone as the whole system.[2] The same discipline is useful here: methodology is not a declaration on a webpage. It is a chain that must survive contact with real evidence.
Five questions to ask about any AI assessment
What exact claim is the assessment designed to support?
Does the task create a credible opportunity for the relevant behaviour to appear?
Can a customer trace each material interpretation through readable evidence references and source-location cues?
Does the report explain unavailable evidence, missing ratings and the resulting limitations?
Does the report make clear that population benchmarks are currently unavailable?
A trustworthy five-page report should let a customer understand what was measured, what the recorded work supports, which readable evidence references and source-location cues support material claims, why a rating is missing, and the limits on interpretation. Report hashes support publication integrity. Raw AI transcripts, private notes, full event logs, full source text and the complete internal evidence package are not exposed by default. The score is useful only when the visible evidence chain supports it.
Sources
[1]AI RMF Core, National Institute of Standards and Technology (26 January 2023).
