Check when an AI assistant's account of a tool call disagrees with the tool result

Identify the disagreement before repeating the action

Imagine a synthetic operations team preparing a weekly handoff. A worker asks an assistant to add a note to a shared case and notify the case owner. The assistant replies, “The case has been updated and the owner notified.” The case-update tool returns a case identifier and a successful version number. The notification tool, however, returns 429 ratelimited and no message identifier. This is one completed action and one uncompleted action, not a completed handoff. Retrying both calls might duplicate the case note while still failing to notify the owner.

The worker should capture the raw responses before relying on the assistant's summary. Record each operation independently: request identifier, target, authorized action, tool status, object identifier, version or timestamp, and any error. An error message is evidence about that attempt; it is not proof that no side effect occurred if the system timed out after accepting the request. Conversely, an assistant's fluent confirmation is not a receipt from the destination. Where a tool returned ambiguous status, check the destination using an independent read before retrying a non-idempotent action.

The OWASP Excessive Agency guidance describes how agent tools can perform consequential actions and recommends downstream authorization, logging, and monitoring. Its focus is security and agency, not a claim that every tool call has a retrievable receipt. The NIST AI RMF Measure playbook suggests histories, audit logs, and human oversight to examine errors and decisions. These are voluntary risk-management suggestions; the reconciliation record below is our own practical method for a knowledge-work handoff.

Sources: [1] [2]

Reconcile the action, the response, and the destination

The fictional handoff record has two rows. In the first: requested = append the approved note to case C-204; tool response = success, version 17, note ID N-81; destination read = case C-204 contains N-81; disposition = complete. In the second: requested = notify the owner of C-204; tool response = rate limited, no message ID; destination read = no matching notification in the approved channel as of the check time; disposition = not confirmed, escalate or retry only the notification under the team's retry rule. The assistant's original sentence is corrected to “The case note was saved; owner notification has not been confirmed.”

Use a stable correlation key when the tools support one, so a retry can be matched to the original request. If the destination does not expose a readback, do not fabricate a success state. Mark it “unverified,” name who can inspect the channel, and delay an external claim until that person checks. Some systems may accept a request and update their state asynchronously, so the check time and expected processing window matter. A status code, log line, and destination view can disagree for legitimate timing reasons; preserve the disagreement rather than collapsing it into a binary success label.

This is narrower than designing an entire automation before launch. The review starts after a particular assistant-tool interaction and asks whether the assistant's account matches the evidence for each attempted action. A downstream API may also return content that looks like instructions for the assistant. Treat that content as data from a lower-trust source, not as permission to expand the action. OWASP's output-handling guidance concerns validation of model output before it reaches other systems; the reconciliation check adds a separate comparison against what the tool actually did.

Sources: [3]

Release a truthful handoff and retain the evidence

Before reporting completion, compare the final sentence with every material row. If a record is saved but a message is pending, state both. If a tool returned success but the destination shows the wrong target, stop and investigate the target mismatch. If an action cannot be read back, distinguish “requested” from “confirmed.” Give the next owner the case identifier, pending operation, error, check time, and permitted retry or escalation route. Never ask the assistant to infer completion merely from its own earlier wording.

A team can test this method with synthetic cases: full success, partial success, timeout after an unknown side effect, stale readback, wrong target, and duplicate retry. The test is whether a reviewer can find the true state and write an accurate handoff without causing a second side effect. It does not establish that a live integration is safe or that all errors are observable. In a Two Prune assessment, a participant can reconcile synthetic tool responses and a case record; the submitted handoff and available workflow evidence can inform human interpretation of task-bounded AI fluency. Two Prune does not operate the customer's live tools or certify their controls.