Verifying hidden dependencies in AI-enabled knowledge work

Reconstruct the path behind the output

Start from one decision artifact and work backward. Identify who requested it, which sources were available, how they were retrieved, which model and orchestration steps were used, what tools or calculations ran, where intermediate data was stored, which identities acted, and who reviewed the result. Record defaults that users may not see.

Classify dependencies as data, model, prompt or instruction, software library, connector, identity, configuration, infrastructure, external service, human approval, or policy assumption. Mark whether each is declared, inferred, testable, replaceable, monitored, and owned. A citation to a source does not reveal the retrieval filter or cached version that supplied it.

The NIST Secure Software Development Framework project provides current first-party guidance and profiles for secure software practices, including generative AI development. It supports documenting and managing software components and development dependencies without proving that a particular workflow is secure.

Sources: [1]

Build the dependency exposure map

Use one row per dependency with the output element, component, provider, version, configuration, identity, data boundary, upstream source, transformation, owner, failure mode, test, observed evidence, fallback, monitoring event, and change trigger. Link dependencies that share a common service or credential so one failure is not counted as several independent safeguards.

Test both presence and substitution. Remove a connector, revoke a permission, change a source version, disable a tool, or route a request through the approved fallback. Observe whether the workflow fails visibly, continues with stale information, or silently uses a different path. Preserve the configuration and timestamp for every result.

The NIST Generative AI Profile discusses risks across the AI lifecycle, third-party components, data, information integrity, security, and monitoring. The profile is voluntary guidance and does not enumerate every dependency in a real system.

Sources: [2]

Worked map: a synthetic board metric

A synthetic board note states that customer response time improved after an AI assistant launch. The dependency map traces the chart to a spreadsheet export, a timezone conversion library, an analytics filter, a cached customer-status table, and an AI-generated narrative. The final note mentions only the spreadsheet.

Review shows that the cached status table excludes recently churned accounts and the conversion library uses a default timezone that shifts several tickets across the reporting boundary. The team refreshes the table, fixes the timezone, reruns the calculation, and narrows the narrative. It records that the apparent improvement is not stable under the corrected population.

The fallback test disables the analytics connector. Instead of stopping, the workflow retrieves a previous export with the same filename. The team changes the process to fail visibly when source date and hash do not match the request. This original example is synthetic and proves nothing about a real deployment or outcome.

Challenge change, failure, and ownership

Ask an independent reviewer to reproduce the artifact using the recorded dependency set. Then change risk-selected components one at a time and test whether monitoring, error messages, and fallback behavior expose the difference. Review shared credentials, broad service accounts, stale caches, copied formulas, hidden model fallbacks, and external calls.

Reopen verification when a provider, model, library, connector, identity, data source, policy, or environment changes. Report unresolved and unobservable dependencies separately from failed tests. A clean final document is not evidence that the path behind it is controlled.

Two Prune can observe whether a participant identifies and tests hidden dependencies inside a bounded simulation. It does not operate software controls, certify security or reliability, provide legal conclusions, or make employment decisions.