Incident response for AI-enabled knowledge work

Define the event before assigning severity

Describe the observed outcome, affected people or process, time window, AI contribution, downstream action, and evidence currently available. Separate an AI-originated failure from a conventional data, access, workflow, or human-review failure that happened around an AI system. Do not wait for perfect classification before containing a material risk.

The OECD AI Incidents and Hazards Monitor methodology distinguishes events involving actual harm from circumstances that could plausibly lead to harm. Its public monitor is necessarily incomplete and does not replace an organization’s own reporting criteria, but the incident-versus-hazard distinction is useful for triage.

Use local severity criteria tied to consequence, exposure, reversibility, affected population, and continuing risk. A fabricated citation caught in a private draft may be a quality event. The same behavior in an externally issued safety instruction could require a much stronger response. Legal classification and notification duties require jurisdiction-specific advice.

Sources: [1]

Build the dual-track incident card

The containment track records the current owner, stop action, affected output or workflow, access changes, customer or worker protection, communications authority, and conditions for restoring service. The learning track records the first report, original output, prompt or configuration where available, source inputs, model and version, human actions, timestamps, expected behavior, competing explanations, and unresolved evidence.

Freeze relevant artifacts without copying sensitive data into an uncontrolled incident document. Use references to governed evidence stores, apply access limits, and preserve a chain of custody appropriate to the context. Document what is missing. A reconstructed prompt is not the original interaction, and a screenshot may omit retrieval or system context.

The final NIST Generative AI Profile recommends documenting and sharing relevant generative-AI incidents, maintaining change records and version history, and clarifying actor responsibilities. It is voluntary risk-management guidance rather than a universal reporting rule.

Sources: [2]

Worked artifact: a synthetic customer notice

Imagine a synthetic customer-operations workflow where an assistant drafts renewal notices from account records. One notice states that service will end in seven days, although the signed amendment provides thirty days. A reviewer sends the message before checking the agreement.

The containment track pauses automated drafting for renewal notices, identifies all messages produced from the same configuration, assigns a customer-communication owner, and prevents unsupervised sends. It does not shut down unrelated internal brainstorming uses. The learning track preserves the sent notice, agreement version, retrieved account fields, assistant output, review action, and release log.

Initial evidence suggests two failures: retrieval selected a stale contract summary, and the interface did not surface the signed amendment. The review procedure also relied on a generic “looks correct” checkbox. The team labels the event a confirmed operational incident under its internal criteria but makes no unsupported legal conclusion.

Before restoration, the owner tests conflicting-document cases, requires the signed agreement to be displayed beside the draft, blocks release when contract authority is unresolved, and confirms that the stop control works. The incident card records each test and its disposition. The scenario is synthetic and does not prove that these controls would be sufficient in another organization.

Investigate causes without forcing one story

Maintain a hypothesis table. Possible contributors include stale data, retrieval failure, ambiguous task definition, model output, interface design, missing reviewer capacity, incentive pressure, permissions, or an unauthorized workflow change. For each hypothesis, list supporting evidence, contradictory evidence, and the next discriminating test.

Avoid assigning the entire cause to the person who clicked release or to “hallucination” without examining the operating system around the output. Equally, do not excuse a preventable human action because AI was involved. The aim is to identify conditions that can be changed and verified.

Verify recovery and retain learning

Re-run representative and adverse cases after repair. Confirm the source hierarchy, stop mechanism, reviewer authority, logging, and escalation route under realistic time pressure. Monitor for recurrence and unintended effects. Close the event only when the accountable owner accepts residual uncertainty and records why operation may resume.

This method supports disciplined containment and learning. It does not certify cybersecurity, determine regulatory reporting, or ensure prevention. Two Prune can observe incident-related judgment and verification in bounded simulations; it does not operate customer incident response.