Practical guide
Can managers assess AI capability by observation?
Yes, but only in a bounded and structured way. A manager can observe how someone frames a realistic task, uses AI, checks evidence, responds to uncertainty, and delivers an artifact. Ordinary workplace observation alone is incomplete: important verification may be invisible, people receive unequal opportunities, task difficulty varies, and a polished output can hide weak reasoning. Use.
Yes, but only in a bounded and structured way. A manager can observe how someone frames a realistic task, uses AI, checks evidence, responds to uncertainty, and delivers an artifact. Ordinary workplace observation alone is incomplete: important verification may be invisible, people receive unequal opportunities, task difficulty varies, and a polished output can hide weak reasoning. Use predefined criteria, inspectable evidence, and corroborating tasks.
Decide what observation can answer
Observation can support a narrow question such as: “During this representative vendor-comparison task, did the participant verify AI-supported claims against supplied sources and escalate a material conflict?” It cannot by itself establish a stable, universal level of AI capability, explain performance in every context, or show that training caused a change.
Write the target behavior, conditions, expected standard, and evidence before watching the task. The CDC’s learning-objective guidance uses audience, behavior, condition, and degree to make expected performance explicit. That is useful for designing an observable task because it prevents an assessor from substituting a vague impression for the intended behavior. CDC: Design training — learning objectives
Sources: [1]
Use the WATCH observation protocol
WATCH converts “I watched them work” into a reviewable assessment record.
### W — Work sample and decision
Choose a representative task with a real output, relevant evidence, ambiguity, and a consequential decision. State what the observation is intended to support. A task that only asks someone to write a prompt cannot reveal evidence navigation, verification, or judgment.
Include at least one designed difficulty: a stale source, conflicting instruction, calculation trap, unsupported AI claim, or unclear authority. The participant needs a genuine opportunity to demonstrate the target behavior.
### A — Anchors before observation
Define behavioral anchors for each criterion before seeing performance. For verification, anchors might distinguish:
Keep communication quality separate from source use. Fluent prose should not raise an evidence score, and visible hesitation should not lower it unless the criterion concerns time or escalation.
### T — Trace visible and hidden work
Observe actions, but also inspect the artifacts that make hidden work reviewable: sources opened, prompts and responses where permitted, notes, calculations, revisions, citations, escalation records, and the final deliverable. Ask a short evidence-focused follow-up: “Which source governed this claim?” or “What made you reject the first answer?”
NIST’s AI Risk Management Framework calls for operator proficiency and human oversight processes to be defined, assessed, and documented. Its playbook also recommends histories, audit logs, override data, errors, complaints, and adjudication records where appropriate. These are useful evidence prompts, not a universal scoring instrument. NIST AI RMF Core and NIST AI RMF Measure Playbook
### C — Conditions and comparison
Record the task version, available sources, tool and model, time limit, permitted assistance, prior practice, accessibility adjustments, interruptions, and observer involvement. If two people receive different cases or support, do not treat their scores as directly comparable without examining that difference.
Use more than one observation when the decision warrants it. One familiar task may overstate capability; one unusually difficult case may understate it. A second task should vary the surface details while preserving the underlying behavior to be demonstrated.
### H — Human review and corroboration
Train reviewers on the anchors, have them score evidence before discussing the participant, and adjudicate material disagreements. Corroborate observation with the submitted artifact and evidence trail. For learning evaluation, compare like-for-like tasks before and after only when conditions are sufficiently comparable.
CDC’s Quality Training Standards list observed practice, exercises, and problem solving as learner-assessment options and recommend realistic, contextually relevant scenarios with feedback. CDC’s measurement guidance also distinguishes learning from transfer and cautions that postcourse reactions do not objectively assess either. CDC Quality Training Standards and CDC: Measuring training effectiveness
- insufficient evidence: accepts a material claim without checking it;
- partial evidence: checks one source but misses a version conflict;
- sufficient evidence: traces the claim to the current authoritative source and records the result; and
- strong evidence: also identifies the conflict’s consequence and uses the correct escalation route.
Sources: [2] [3] [4] [5]
Worked observation: a vendor-comparison memo
A synthetic assessment asks a participant to recommend between two software vendors. The packet contains current price sheets, a security questionnaire, a service-level policy, meeting notes, and an older comparison table. The AI assistant confidently states that Vendor B has the shorter recovery objective, but the current questionnaire contradicts it.
| Criterion | Observable evidence | Hidden or artifact evidence | Disposition | | --- | --- | --- | --- | | Task framing | Participant restates decision, audience, cost horizon, and non-negotiable controls | Notes list decision criteria before AI use | Sufficient | | Evidence navigation | Opens both questionnaires and the old comparison | Source list records version dates | Strong | | AI orchestration | Requests a table with citations, then narrows the comparison | Prompt history shows the assistant was not asked to choose the vendor | Sufficient | | Verification | Challenges the recovery claim after reading the current questionnaire | Memo cites the current field and marks old table superseded | Strong | | Judgment | Explains tradeoff between price and recovery evidence | Decision log names residual uncertainty in implementation capacity | Sufficient | | Communication | Delivers a concise recommendation with next action | Final memo separates facts, inference, and unanswered question | Sufficient |
The observer also records one limitation: the task provided well-organized sources, so it offers little evidence about finding material across a fragmented workspace. The next task should test retrieval under less curated conditions. The report does not convert this single observation into a broad claim about all AI-enabled work.
Common observation failures
### Watching only the final minutes
The visible deliverable hides discarded sources, AI overreliance, verification, and revision. Observe or preserve the full bounded process where proportionate.
### Letting normal workload determine the assessment
Some people encounter rich AI tasks while others do not. Unequal opportunity makes passive managerial observation a weak comparison. Use a common representative simulation when comparable evidence is needed.
### Scoring style as capability
Confidence, speed, verbosity, and presentation fluency can influence an observer without proving source use or judgment. Anchor every score to a behavior and artifact.
### Treating surveillance as evidence quality
More monitoring is not automatically better. Collect only evidence needed for the assessment purpose, disclose the conditions, protect access, and distinguish assessment evidence from routine employee monitoring. Observation design should respect applicable policy, privacy, and worker-representation requirements.
### Ignoring observer effects and disagreement
People may change behavior when watched, and managers may bring prior impressions. Use standardized instructions, evidence-first scoring, reviewer calibration, and an appeal or clarification route where appropriate.
When observation is not enough
Add another method when the behavior is rare, dangerous to provoke, mostly cognitive, dependent on confidential cases, or not observable within the available period. A simulation, evidence-based interview, artifact review, knowledge check, or repeated work sample may provide the missing opportunity. Delayed workplace evidence may help examine transfer, but it should not be confused with the controlled assessment score.
Two Prune’s boundary
Two Prune creates realistic AI-enabled work simulations and records evidence such as source navigation, AI interactions, verification behavior, revisions, and submitted artifacts. Managers can use the assessment report as structured evidence for human interpretation. Two Prune does not continuously monitor employees, infer capability from casual observation, predict all future performance, or make an employment decision.
Sources and limits
These sources inform the observation design. They do not validate WATCH or establish that one observed task is sufficient for a particular organizational decision. The worked case is synthetic.
- CDC: Design training — learning objectives — defining audience, behavior, conditions, and degree.
- CDC Quality Training Standards — realistic learner assessment through exercises, observed practice, and problem solving.
- CDC: Measuring training effectiveness — distinctions among learning, learning transfer, and postcourse evaluation.
- NIST AI RMF Core and Measure Playbook — operator proficiency, oversight, documentation, logs, overrides, errors, and adjudication evidence.
Sources: [1] [4] [5] [2] [3]
Sources
- 1.CDC: Design training — learning objectives · Centers for Disease Control and Prevention
- 2.NIST AI RMF Core · National Institute of Standards and Technology
- 3.NIST AI RMF Measure Playbook · National Institute of Standards and Technology
- 4.CDC Quality Training Standards · Centers for Disease Control and Prevention
- 5.CDC: Measuring training effectiveness · Centers for Disease Control and Prevention