Practical guide

How to assess whether someone can handle AI overreliance

Assessing AI overreliance requires a task in which accepting a plausible output creates a detectable consequence. Frequency of AI use, confidence, a policy quiz, or one polished deliverable cannot show whether a person calibrated reliance to evidence and stakes. Observe what they check, reject, revise, escalate, and preserve when AI output conflicts with the record. This method uses BRAKE:.

By Two Prune

Assessing AI overreliance requires a task in which accepting a plausible output creates a detectable consequence. Frequency of AI use, confidence, a policy quiz, or one polished deliverable cannot show whether a person calibrated reliance to evidence and stakes. Observe what they check, reject, revise, escalate, and preserve when AI output conflicts with the record. This method uses BRAKE: bounded task, reliance traps, actions observed, knowledge check, and evidence-based disposition.

Source basis and limits

NIST's Generative AI Profile identifies automation bias and excessive deference to AI output as human-AI configuration risks. It recommends context-sensitive oversight, tracking, documentation, testing, incident handling, and evidence that can be linked to original sources. The AI RMF Core calls for clearly defined human-AI roles, documented knowledge limits, and repeatable testing and evaluation. These voluntary sources support a risk-based assessment design; they do not define a universal person-level score or remove the need for local domain review. NIST AI 600-1 NIST AI RMF Core

Sources: [1] [2]

B — Bound the task and consequence

Choose a representative task with a real decision, sufficient source material, and a consequence that matters but is safe to simulate. Document:

| Design field | Required entry | | --- | --- | | Task and audience | | | Decision and deliverable | | | Approved AI role | | | Sources available | | | Error consequences | | | Checks expected | | | Escalation authority | | | Time and tool constraints | |

The task should make calibrated reliance possible. If no authoritative source, calculation route, or escalation path is available, failure to verify may reflect poor assessment design rather than participant behavior.

R — Plant reliance traps

Use a small number of disclosed-to-reviewers, undisclosed-to-participants traps that arise naturally from the work:

Every trap needs an evidence key: why it is wrong or incomplete, what clues are available, which reasonable checks could expose it, and what acceptable responses look like. Avoid trivia, obscure facts, and traps that depend on guessing the designer's intent.

  • a fluent claim contradicted by a supplied source;
  • a real source cited for a proposition it does not support;
  • a correct total built from the wrong denominator;
  • a stale policy presented as current;
  • an omitted high-consequence exception hidden by an accurate summary;
  • an instruction that exceeds the participant's authority;
  • an answer that should trigger specialist review rather than confident completion.

A — Observe actions, not just the final answer

Capture a minimal process record:

| Signal | Strong evidence | Ambiguous evidence | Risk evidence | | --- | --- | --- | --- | | Source use | Opens original source and maps claim to passage | Opens source without using it | Cites inaccessible or mismatched source | | Calculation | Reproduces result with units and denominator | Notices concern but does not resolve it | Accepts confident number without checking | | AI challenge | Tests a claim or asks for uncertainty and alternatives | Re-prompts for style only | Treats fluency as authority | | Revision | Corrects output and records reason | Silently edits with unclear basis | Preserves known error | | Escalation | Identifies authority limit and routes issue | Notes risk without owner or trigger | Makes specialist decision without authority | | Delivery | Shows residual uncertainty and consequence | Uses generic caveat | Suppresses material exception |

Do not infer reasoning solely from absence of visible activity. A participant may know a fact, use an unavailable external source, or perform a mental check. The assessment should ask for a brief verification record and mark genuinely missing evidence as unavailable.

K — Add a short knowledge and transfer check

After the task, ask the participant to explain:

Then vary one condition: increase the stakes, remove a source, change the audience, or provide conflicting expert input. A participant who memorized “always verify” but cannot adapt the verification route has not yet shown calibrated reliance.

  • Which AI output did you trust least, and why?
  • Which check changed your deliverable?
  • What remains uncertain?
  • When would the same check be insufficient?
  • What would you escalate, to whom, and with what evidence?

E — Make an evidence-based disposition

Use behaviorally anchored dispositions rather than a personality label:

Report the specific behavior, source, consequence, and limit. Do not generalize from one simulation to every role, tool, domain, or future situation.

  • Calibrated in this task: checks matched the consequence, material errors were repaired or escalated, and uncertainty remained visible.
  • Partially evidenced: some appropriate checks occurred, but important behavior was missing, inconsistent, or unobservable.
  • Overreliance risk observed: the participant accepted consequential AI output despite available contradictory evidence or an expected check.
  • Underreliance risk observed: the participant rejected useful AI support without considering evidence or comparative quality.
  • Not interpretable: the task, tooling, sources, time, accessibility, or event capture prevented a fair observation.

Worked example: fictional supplier-risk briefing

A participant must recommend whether a fictional team should continue a supplier pilot. The source pack includes an incident log, contract excerpt, usage table, and accessibility review. The AI workspace produces a polished briefing that says incidents fell by 40%, cites the incident log, and recommends expansion.

The figure is mathematically correct only because the later period contains fewer active users. Per 100 active users, incidents increased. The cited log records counts but does not support the causal claim that the tool improved. A contract clause also requires security approval before the proposed data use, and the accessibility review has an unresolved keyboard blocker.

A calibrated response reopens the usage table, normalizes the rates, separates count change from causal inference, and rejects the expansion recommendation. It preserves the accessibility exception in the executive summary and routes the security question to the named owner. The final disposition supports only a paused, bounded review with specific evidence needed to continue.

An ambiguous response deletes the 40% sentence but leaves no calculation or reason. An overreliance signal appears when the participant retains the expansion recommendation after opening the contradictory table. An underreliance signal would be refusing all AI-generated structure even after verifying it, while failing to complete the task. The reviewer records each observation and does not turn one behavior into a permanent trait.

Assessment QA checklist

[ ] The task, AI role, sources, consequence, checks, and authority are explicit.

  • [ ] Each trap has a reviewer evidence key and more than one reasonable detection route.
  • [ ] Process evidence captures source use, calculations, challenge, revision, escalation, and delivery.
  • [ ] Accessibility and tool constraints are tested before use.
  • [ ] Missing activity is not automatically scored as missing reasoning.
  • [ ] The follow-up varies context to test adaptation.
  • [ ] Reviewers use anchored dispositions and retain disagreement.
  • [ ] Reporting stays task-specific and states interpretation limits.

Two Prune boundary

Two Prune can present a realistic simulation, retain relevant workspace evidence, and report observed performance across context framing, evidence navigation, AI orchestration, verification and risk control, judgment and synthesis, and communication and delivery. It can show where overreliance-related behavior appeared in that assessment. It does not diagnose a person, predict all future behavior, certify safe AI use, make employment decisions, or replace domain, accessibility, legal, security, or psychometric review.

Sources

NIST AI 600-1: Generative AI Profile

  • NIST: AI Risk Management Framework Core

Sources: [1] [2]

Sources

  1. 1.NIST AI 600-1 · doi.org
  2. 2.NIST AI RMF Core · National Institute of Standards and Technology