A Quiz Can Test Whether Someone Knows AI Can Be Wrong. A Simulation Shows What They Do When It Is.
Knowledge tests can check concepts. Realistic work simulations can reveal how someone responds when evidence conflicts, time is limited and a persuasive AI answer needs to be challenged.
Two Prune Research··4 min read
A knowledge test can ask whether AI may fabricate a source. A simulation can capture whether the recorded work identifies the fabricated source before it reaches the final deliverable.
Consider an operations lead asked whether to automate a customer-refund workflow. The source pack includes process data, a policy excerpt, a stakeholder email and a vendor estimate. Two figures use different reporting windows. The vendor claims a large saving. One stakeholder asks for speed while another flags a control risk.
A quiz can ask for the definitions of hallucination, privacy and human oversight. It cannot show whether the recorded work reconciles the reporting periods, uses the policy constraint, gives AI relevant context, challenges the unsupported saving, proposes a proportionate control and produces a decision memo that makes the trade-off clear.
The difference is not difficulty. The two formats observe different things.
Knowledge and performance are different questions
Knowledge tests are efficient when the question is whether someone understands a concept, policy or risk. Self-assessments can reveal confidence and attitudes. Usage analytics can show whether a tool was opened. None of those measures is useless.
But workplace AI use is a system of interactions among a person, a task, source material, organisational constraints and an AI tool. NIST's sociotechnical evaluation work emphasises assessing AI in context, including what happens when people interact with it in realistic settings.[1] If the question is how knowledge becomes work, the assessment needs work to observe.
What a realistic simulation makes visible
Whether the recorded work addresses the audience and decision before content is generated.
Which sources are opened or used when documents conflict or contain distracting detail.
What work is completed with and without AI assistance.
Whether calculations, citations and recommendations are tested against the evidence.
How uncertainty, sensitive information and operational consequences are addressed in the recorded work.
Whether the final deliverable helps its intended audience make the decision.
A simulation also allows process and outcome to disagree. A polished answer may be paired with a fragile recorded process. Another recorded sequence may include a correction after an AI error appears. These observations do not establish intent, causation or stable traits, and the final document alone cannot describe the recorded difference.
Put the framework to work
See how your workforce demonstrates AI fluency.
Two Prune uses realistic work simulations to produce structured assessment evidence—not automated employment decisions.
Two Prune is beginning a research programme—and inviting academic collaboration—to develop task-bounded AI-fluency assessment into a responsible measurement method.
The scenario should resemble the decision, not copy the job
Realism does not require production data or a perfect replica of someone's role. A useful scenario can use synthetic or explicitly approved non-sensitive materials. What matters is structural similarity: a credible objective, evidence that rewards careful navigation, constraints that affect the recommendation and an audience that changes how the result should be communicated.
The strongest scenario is not the one with the most documents or hidden traps. It is the one that gives the intended behaviours a fair chance to appear. If the assessment is meant to examine verification, there should be something material to verify. If it is meant to examine judgment, the task should contain a genuine trade-off rather than one obviously correct answer.
The limitations belong in the report
A simulation is still a sample of behaviour. Performance can vary by scenario, domain familiarity, accessibility needs, time limit and tool familiarity. Activity evidence can show that an action occurred without fully revealing the participant's reasoning. NIST's risk framework likewise calls for context-aware measurement and documentation of uncertainty.[2] A strong report therefore uses only available evidence, explains missing ratings and states explicit limitations. If evidence remains materially insufficient after bounded recovery, the result is a score-free limited-evidence report, not a lower score. Population benchmarks remain unavailable until a sufficiently mature, compatible comparison group exists.
Two Prune does not claim that one simulation predicts universal workplace performance or replaces human judgment. Its advantage is narrower: a realistic scenario can produce direct, task-bounded evidence of applied AI fluency that a knowledge-only test is not designed to capture.
Choose the format after you choose the question
Use a quiz when you need to check knowledge. Use a survey when confidence or attitudes are the subject. Use activity data when recorded tool activity is the question. Use a realistic work simulation when you need task-bounded evidence of how the recorded work frames, investigates, verifies, judges and delivers a complete AI-enabled task.
