Practical guide
Evidence that prompt-injection defenses are operating
A prompt-injection policy, system prompt, or blocked demonstration does not show that defenses operate across a live AI workflow. Operating evidence must cover direct and indirect inputs, privileged actions, data exposure, tool calls, monitoring, human escalation, exceptions, and changes over time. The defensible conclusion is bounded: controls operated as tested for a named system version,.
A prompt-injection policy, system prompt, or blocked demonstration does not show that defenses operate across a live AI workflow. Operating evidence must cover direct and indirect inputs, privileged actions, data exposure, tool calls, monitoring, human escalation, exceptions, and changes over time. The defensible conclusion is bounded: controls operated as tested for a named system version, threat set, population, and period.
Security and assurance boundary
NIST defines prompt injection as exploiting the combination of untrusted input with a prompt constructed by a higher-trust party. Its Generative AI Profile distinguishes direct attacks from indirect attacks delivered through retrieved data and recommends risk management across design, development, deployment, and use. The UK's National Cyber Security Centre secure-AI guidance emphasizes secure design, development, deployment, and operation across the system lifecycle. These sources support testing and evidence discipline; they do not prescribe one complete defense or certify that a system is secure. NIST prompt-injection glossary NIST AI 600-1 Generative AI Profile NCSC Guidelines for secure AI system development
Sources: [1] [2] [3]
Use the SHIELD evidence protocol
### S — Scope the system and trust boundaries
Record the application, model and version, system instructions, retrieval sources, tools, permissions, identities, data classes, output destinations, and human approval points. Draw trust boundaries from input to action. Identify where untrusted content can enter through users, documents, webpages, email, connectors, tool results, memory, or model outputs.
State the control objective in observable terms: for example, untrusted document text must not cause the application to reveal restricted context or execute a high-impact action without an authorized approval.
### H — Hold a versioned threat set
Maintain a test set covering direct instruction override, indirect instructions in every supported content type, encoded and obfuscated variants, multi-turn attacks, data exfiltration attempts, tool misuse, cross-context contamination, and benign lookalikes. Link each case to a threat hypothesis, expected result, severity, and affected boundary.
Do not report only attack strings known to the development team. Include independent cases and preserve prior failures as regression tests. Track coverage gaps explicitly.
### I — Instrument decisions and effects
Capture input provenance, retrieval events, policy and classifier decisions, model and prompt versions, tool requests, authorization results, user approvals, outputs, blocked effects, and incident links. Redact or minimize sensitive content while retaining enough evidence to reconstruct the path.
A blocked final response is insufficient if a privileged tool was already invoked. Verify effects at the authorization and destination layers.
### E — Execute adversarial and benign tests
Run the threat set in an isolated environment that mirrors material production controls. Test expected attacks and benign content so false positives are visible. Reconcile attempted cases, executed cases, valid results, failures, inconclusive results, and exclusions.
Use defense in depth: constrain privileges, separate instructions from untrusted data, validate tool inputs, require authorization for consequential actions, filter outputs where appropriate, and monitor behavior. Treat model-level resistance as one layer, not the control boundary.
### L — Link failures, exceptions, and incidents
For every failure, record the observed path, potential impact, affected versions, containment, owner, due date, residual risk, and retest. Reconcile test failures with production alerts and user-reported events. A caught attack demonstrates detection, not absence of attack. An empty incident register is not proof that the control works.
### D — Decide, expire, and retest
Issue operating as tested, operating with limitations, not operating as intended, or insufficient evidence. Name the tested period and population. Expire evidence after material changes to models, prompts, retrieval, tools, permissions, data, monitoring, or threat intelligence. Retest repaired failures and representative unaffected paths.
Minimum evidence record
| Evidence | Required fields | Check | | --- | --- | --- | | system boundary | versions, components, data, tools, permissions, approvals | architecture matches tested deployment | | control map | threat, preventive, detective and response controls, owner | each critical effect has an independent authorization boundary | | test manifest | case ID, provenance, expected result, severity, version | manifest is frozen before execution | | execution log | case, result, trace, effect, timestamp, evidence ID | totals reconcile with the manifest | | benign set | representative legitimate tasks and expected outcomes | false-positive and task-completion effects are reported | | exception register | failure, exposure, containment, owner, due date, retest | open exceptions appear in the decision view | | operating sample | production alerts, approvals, tool denials, incidents, gaps | sample supports only the stated period and scope | | disposition | conclusion, authority, limitations, expiry, next test | wording matches the evidence |
Worked control test: synthetic research assistant
A fictional research assistant retrieves approved webpages and documents, summarizes them, and can save a note. It cannot send messages or access customer records. The team claims its prompt-injection defense works because 40 direct override prompts were blocked.
SHIELD maps three additional entry paths: instructions embedded in webpages, text hidden in uploaded PDFs, and malicious content returned by the search connector. The versioned set contains 180 adversarial cases and 60 benign documents. Expected effects include refusing restricted-data requests, preventing unauthorized note destinations, preserving citations, and escalating suspicious tool instructions.
The run reconciles 240 cases. Four indirect PDF cases cause the model to include a bogus instruction in its draft, but the note tool's destination allowlist prevents an external write. One connector case causes restricted system text to appear in an internal trace, though not in the user output. Nine benign documents are incorrectly blocked because they discuss prompt injection as a topic.
The control is not operating as intended for the tested version. The allowlist limited one consequence but did not prevent instruction following, and the trace exposure is a separate failure. The owner disables PDF retrieval for the affected path, restricts trace access, adds provenance-aware parsing, tunes the detector against the benign set, and reruns all 240 cases plus new regression tests. The record does not claim the application is secure after containment.
Failure conditions
Reject an “effective” conclusion when the system boundary is incomplete, tests omit indirect paths, only blocked outputs are observed, excluded cases do not reconcile, benign-task impact is absent, critical failures lack containment, or evidence predates a material system change. Hold deployment or restrict capabilities when a plausible attack can cause an unauthorized high-impact effect and no independent control contains it.
Two Prune boundary
Two Prune can assess whether a participant identifies prompt-injection evidence gaps, traces supplied events, tests control claims, and writes a bounded disposition in a simulation. It does not penetration-test or certify an organization's AI system, promise security, provide legal compliance, approve deployment, predict workplace performance, or make employment decisions.
Sources
NIST: Prompt injection glossary
- NIST: Artificial Intelligence Risk Management Framework—Generative AI Profile
- UK NCSC: Guidelines for secure AI system development
Sources: [1] [2] [3]
Sources
- 1.NIST prompt-injection glossary · National Institute of Standards and Technology
- 2.NIST AI 600-1 Generative AI Profile · National Institute of Standards and Technology
- 3.NCSC Guidelines for secure AI system development · www.ncsc.gov.uk