How to Detect Automation Bias in AI-Assisted Work

Map the decision environment

Record the task, user, automation output, source evidence, alternatives, time pressure, consequence, interface cues, and approval route. Ask whether the system recommendation appears before independent review, whether uncertainty is visible, and whether disagreement requires extra effort.

NIST’s AI RMF Core recommends documenting human-AI roles, knowledge limits, oversight, testing, and context-specific risks (AI RMF Core). Its Generative AI Profile provides additional risk-management actions for generative systems (Generative AI Profile). These sources support risk analysis, not a diagnostic label for an individual.

Sources: [1] [2]

Test omission and commission errors

Create scenarios where the automated recommendation is correct, incorrect, incomplete, and absent. Commission error occurs when a person follows a wrong recommendation. Omission error occurs when a person misses a problem because the system did not flag it. Include a defensible disagreement path.

Measure source inspection, alternative generation, contradiction handling, final decision, and confidence. Do not infer thought process from a click alone. Capture limitations and off-system checks.

Worked scenario

Imagine a synthetic contract-review assistant that flags a payment term as standard and leaves a cross-reference unmarked. The supplied agreement shows that the cross-reference changes the trigger date. A reviewer accepts the “standard” label and does not open the referenced schedule.

The analysis identifies interface prominence, vague uncertainty, and a workflow that makes source opening optional. The repair removes the authoritative label, displays the exact clause and dependency, and requires the reviewer to resolve or escalate the cross-reference before approval.

A weak conclusion says the reviewer lacks judgment. The evidence supports a task-specific process failure, not a stable personal trait.

Reduce bias through workflow design

Require independent framing before showing the recommendation for high-consequence tasks. Display source provenance and uncertainty near the suggestion. Make disagreement and escalation easy. Use checklists sparingly and test whether they change behavior or become another automatic action.

Review incidents and near misses to update interfaces, training, task allocation, and controls. Monitor whether interventions add review burden or create new shortcuts.

Design a comparison that separates causes

Compare conditions deliberately. One group may see source evidence before the automated recommendation, another may see the recommendation first, and a third may complete the task without it. Keep task difficulty, time, instructions, and available sources as comparable as practical. Record whether participants actually opened the evidence and whether the interface made alternative action visible.

Interpret differences cautiously. A wrong decision after seeing a recommendation may reflect automation bias, but it may also reflect domain knowledge, ambiguous instructions, fatigue, incentives, or a misleading source. Use interviews or structured debriefs to test explanations without treating retrospective accounts as complete process evidence. Repeat the pattern across varied scenarios before changing a high-consequence workflow, and document where the comparison design cannot isolate a cause.

Review accessibility and experience effects as well. A participant who struggles to open a source, interpret a confidence display, or find the disagreement control has encountered a workflow barrier. Record that barrier separately from reliance behavior. Re-run a repaired interface condition before concluding that instruction alone is the appropriate response.

Verification checklist

The goal is calibrated human-AI collaboration, not proof that automation bias has been eliminated.

• Correct, incorrect, incomplete, and absent recommendations are tested.
• Source access and disagreement routes are visible.
• Commission and omission failures are distinguished.
• Interface and workload context are recorded.
• Missing process evidence limits interpretation.
• Findings inform workflow changes rather than personnel labels.
• High-consequence decisions retain accountable human approval.