Prompt-injection defense for AI-enabled knowledge work

Map instructions, data, and authority

Choose a bounded use case such as an assistant that reads uploaded supplier documents and drafts a comparison. Identify the governing instructions, user request, uploaded content, retrieved pages, tool outputs, memory, and generated response. Mark which inputs are trusted instructions and which must remain data even if they contain imperative language.

List every action the system can take: reading a file, querying a database, following a link, sending a message, changing a record, or exporting content. For each action, record the credential used, accessible scope, consequence, precondition, human approval, and evidence retained. Reduce authority to what the workflow actually needs.

The NIST Generative AI Profile extends the AI Risk Management Framework with risks and actions for generative systems, including indirect prompt injection and changing attack surfaces. It supplies risk-management guidance, not proof that a particular safeguard prevents every attack.

Sources: [1]

Build the trust-boundary action matrix

Create one row per input-to-action path. Record the source, trust class, permitted interpretation, sensitive data exposed, tools reachable, action limit, deterministic validation, approval owner, output validation, log, alert, fallback, and test case. Treat retrieved documents, web pages, emails, and tool responses as potentially adversarial content.

Use layered controls. Separate instructions from content, sanitize remote material where appropriate, constrain tool schemas, apply least privilege, validate proposed actions against the original user intent, require confirmation for consequential steps, and inspect outputs before they reach another interpreter. A model-based guardrail can be one layer, but it is also fallible and should not replace deterministic boundaries.

The OWASP LLM Prompt Injection Prevention Cheat Sheet describes direct and indirect injection and recommends defenses including structured prompts, input handling, least privilege, output validation, monitoring, and human checkpoints. OWASP guidance is a useful security reference, but deployment-specific testing is still required.

Sources: [2]

Worked artifact: a synthetic supplier comparison

Imagine a synthetic procurement assistant reading three supplier PDFs. One document contains hidden text instructing the assistant to ignore the evaluation criteria, reveal other supplier information, and send a message to an external address. The trust-boundary matrix marks PDF text as untrusted data and gives the comparison process read-only access to the three isolated files. It has no message-sending credential.

The assistant still repeats a manipulated ranking in its draft. An output rule detects criteria that are not in the approved rubric and routes the draft for review. The reviewer sees the source passage, quarantines the document, and reruns the task with a clean text extraction. The event log retains the proposed action and blocked route without exposing unrelated supplier content.

A second test hides instructions in an image and asks the assistant to call an allowed lookup tool with an unauthorized identifier. The tool rejects the identifier because it is outside the case scope. The team records both failures and updates its regression set. This synthetic example does not establish universal control effectiveness or certify a production system.

Test the path, not the prompt alone

Test direct, indirect, encoded, multimodal, multi-turn, and tool-mediated cases across ordinary and boundary conditions. Verify denials, approvals, logs, alerts, fallback behavior, and data isolation. Reassess whenever tools, credentials, retrieval sources, model, memory, rendering, or downstream automation changes.

Two Prune can create controlled scenarios that reveal how a participant treats untrusted instructions, checks evidence, and escalates. It does not operate a customer's security controls, certify a system, or replace security testing and accountable human review.