Verifying data leakage risk in AI-enabled knowledge work

Map the data paths and trust boundaries

Choose one workflow, such as an internal assistant answering questions from a document repository. Inventory user input, retrieved content, model context, generated output, conversation history, telemetry, feedback, exports, caches, plugins, downstream applications, administrative views, and provider processing. Mark data classification, owner, permitted users, purpose, residency where relevant, and retention route.

Draw every trust-boundary crossing and identify the enforcement point: identity, document permission, retrieval filter, tenant isolation, output rule, connector scope, or administrative authorization. Do not infer actual product behavior from a marketing label. Verify the current offering, plan, configuration, contract, and operating evidence.

NIST's initial public draft Cybersecurity AI Profile notes that an AI model can be as sensitive as the data available to it and proposes access control, minimization, runtime redaction, output filtering, and pattern detection as leakage considerations. Because the document is a draft community profile, it is guidance for test design rather than evidence that a control is effective.

Sources: [1]

Design an access-path canary test

Create synthetic markers that resemble each protected data class without using real secrets or personal information. Place each marker in a controlled source with known permissions. Record the source, authorized identity, prohibited identity, connector, retrieval path, prompt family, expected behavior, output location, log location, downstream copy, cleanup plan, and test owner.

Test direct requests, paraphrases, summaries, cross-document questions, indirect instructions inside retrieved content, exports, shared conversations, and permission changes. Include attempts by an authorized user and an unauthorized user so the test can distinguish missing data from access enforcement. Verify both output and hidden operational copies.

The NIST Generative AI Profile describes confidentiality, privacy, prompt-injection, and interconnected-system risks in generative AI. The profile is voluntary and cross-sectoral; organizations must tailor controls and tests to their own system and obligations.

Sources: [2]

Worked artifact: a synthetic finance canary

A synthetic internal assistant indexes general operations documents and a restricted finance folder. The test team inserts the marker ORCHID-4821 into one finance-only document. A finance user can retrieve it with the expected citation. A general user asking directly receives no finance result, but a cross-document comparison causes the marker to appear in a generated table.

The workflow fails. Engineers find that the comparison tool retrieves with a service identity before applying user filtering. They move authorization ahead of retrieval, clear derived caches, and rerun direct, paraphrased, comparison, export, and shared-link tests for both identities. The marker no longer crosses the boundary, and authorized access still works.

The team checks telemetry and exported files for residual copies, removes all synthetic fixtures, and records the versions and evidence. This example is synthetic and does not establish that a real deployment is secure.

Verify repair and monitor change

Have an independent reviewer reproduce the failed path and the full regression set. Confirm the identity, permissions, source version, connector, model and orchestration configuration, cache state, output, logs, and cleanup. A blocked answer alone is insufficient if the prohibited data still entered model context or an operational store.

Reopen testing after permission-model changes, connector additions, model or provider updates, new plugins, cache changes, exports, or revised retention settings. Monitor access-denial anomalies and incidents, but do not equate absence of reports with absence of leakage.

Use real sensitive data only under approved security and privacy procedures; synthetic canaries reduce exposure during testing but do not cover every risk. Two Prune can observe whether a participant recognizes and handles data-boundary evidence in a bounded assessment. It does not certify security, operate customer controls, or make employment decisions.