Practical guide
What evidence shows source attribution is operating?
A source-attribution policy can exist while finished work still contains unsupported claims, untraceable calculations, stale sources, or citations that merely resolve to a webpage. Operating evidence shows that the control was applied to an actual population, detected meaningful exceptions, drove correction, and was retested. The useful artifact is a source-attribution operating-evidence packet,.
A source-attribution policy can exist while finished work still contains unsupported claims, untraceable calculations, stale sources, or citations that merely resolve to a webpage. Operating evidence shows that the control was applied to an actual population, detected meaningful exceptions, drove correction, and was retested. The useful artifact is a source-attribution operating-evidence packet, not a screenshot of the policy or a count of links.
Define the control before testing it
Use TRACE: Testable purpose, Record design, Assess operation, Challenge cases, and Escalate exceptions. This method is retrospective. It does not replace a source-attribution policy or the attribution packet used while creating work; it tests whether those controls operated on released outputs.
W3C PROV-O provides a vocabulary for representing entities, activities, agents, and their provenance relationships. NIST Special Publication 800-53A provides a general assessment structure using examination, interview, and testing methods, with depth and coverage tailored to the objective. The NIST Generative AI Profile discusses information-integrity risks and practices involving provenance and source verification. Together they support traceable and testable evidence, but none certifies a particular organization's attribution control or establishes legal compliance. W3C PROV-O NIST SP 800-53A Rev. 5 NIST Generative AI Profile
### T: State a testable control purpose
Write what the control is meant to prevent or detect. A useful statement is:
> Before release, each material external claim, calculation, or asset is connected to an identifiable source or is visibly labelled as inference, assumption, calculation, generated material, or synthetic example. A reviewer verifies that the cited evidence supports the final wording and records exceptions.
Define materiality, the covered outputs, the release point, responsible owner, reviewer, evidence retained, and permitted exception. Separate four properties: the source can be located; its identity and version are known; it supports the stated claim; and an accountable person approved the use. One property does not prove the others.
### R: Record control design and implementation
Collect the approved control statement, current procedure, templates, system configuration, role assignments, training or guidance, change history, and release criteria. Then inspect whether people can actually perform the control. A required field that accepts “N/A” without reason may exist in design but fail in implementation. A browser extension that saves URLs may capture location without capturing the supporting passage or version.
Create a design-to-evidence map:
| Control element | Expected operating evidence | Failure example | | --- | --- | --- | | source identity | publisher, title, version or date, stable location | homepage only | | claim support | relevant passage, cells, or calculation inputs | link is related but does not support wording | | transformation | summary, extraction, formula, or generated contribution | calculation cannot be reproduced | | review | reviewer, date, disposition, correction | checkbox with no sampled evidence | | exception | reason, owner, consequence, due date, retest | unresolved source silently released |
### A: Assess a defined operating population
Identify the population of released outputs for a stated period and explain how it was obtained. Test completeness before sampling: reconcile release logs, repositories, publication records, or other authoritative records. Select samples based on risk as well as coverage. Include quantitative claims, consequential recommendations, AI-assisted summaries, reused content, calculations, images, and outputs that crossed format or system boundaries.
For each sample, reconstruct the claim-to-source path. Resolve the retained source, verify the version and date, locate the supporting passage or input cells, reproduce transformations, inspect reviewer evidence, and compare approved wording with the released output. Track broken links separately from unsupported claims. A live link can support the wrong claim; a temporarily unavailable link can still have a valid retained source snapshot.
### C: Challenge positive and negative cases
Do not test only well-formed examples. Seed or select cases that should fail: a relevant source that does not support the exact wording, a superseded release, a secondary summary where a primary record is required, a redirected URL, an unlabeled synthetic asset, a calculation with missing units, and a citation added after review.
Also test control bypasses. Copy an approved paragraph into a new audience or period, export it to another format, refresh a dataset, remove metadata, or change a quantitative qualifier. The control should reopen when the supporting relationship changes. If it passes every case, the test may be too weak.
Record the expected result, observed result, evidence ID, and whether the control prevented release, detected the issue before release, or detected it only afterward. Detection timing matters.
### E: Escalate exceptions and retest
Classify exceptions by their consequence: attribution metadata incomplete, source unavailable, source superseded, claim only partially supported, claim unsupported, calculation irreproducible, generated element unlabeled, or review evidence absent. Name an owner, affected outputs, interim control, due date, and release consequence.
Retest the corrected output and the control that allowed the failure. Fixing one citation does not show that the population-level control now operates. If the same failure can recur because a template, integration, instruction, or reviewer practice is defective, repair that mechanism and sample again. Report unresolved exceptions and limitations to the accountable owner.
Sources: [1] [2] [3]
Operating-evidence record
| Test ID | Output and claim | Source and version | Procedure | Expected result | Observed result | Exception | Retest | | --- | --- | --- | --- | --- | --- | --- | --- | | | | | | | | | |
Summarize the period, population, sample rationale, design evidence, operating tests, exception rate by type, consequences, corrective actions, retests, reviewer, and review date. Avoid a universal “effective” label when coverage is bounded. A defensible conclusion is closer to: the stated control operated for the tested outputs and period, with these exceptions and limitations.
Worked test: a synthetic AI-assisted policy brief
A synthetic team releases 24 policy briefs during a quarter. The control requires material claims to link to a retained source passage and requires an independent reviewer before release. The tester reconciles all 24 briefs to the release log and selects six: two with quantitative claims, two with AI-assisted summaries, one reused from a prior quarter, and one with an image.
The sample includes four seeded challenge cases. One URL redirects to the current official release, but the brief quotes the prior period. One secondary article discusses the topic yet does not support the claimed percentage. One calculation cites the correct dataset but omits the denominator and unit. The image has provenance metadata but is not labelled as a synthetic illustration.
The control detects the missing unit before release and the owner repairs it. It misses the outdated statistic because the reviewer checks only that the redirected page loads. It also misses the unsupported percentage because the workflow stores the URL but not the relevant passage. The image's provenance record shows its editing history but does not establish that the depicted event occurred.
The disposition is operated with material exceptions. The owner adds a retained passage and source date to the review form, requires a claim-support disposition, labels the image as synthetic, corrects the two briefs, and searches the remaining population for the same sources. A retest using one supported claim, one unsupported relevant link, and one superseded release produces the expected results. The packet does not claim that all future outputs are accurate or that the control establishes compliance.
Two Prune boundary
Two Prune can assess whether a participant traces sources, distinguishes provenance from claim support, tests attribution evidence, identifies exceptions, and communicates a bounded conclusion in a realistic simulation. It does not audit or certify an organization's controls, determine legal compliance, establish factual truth across a domain, predict workplace performance, or make employment decisions.
Sources
W3C: PROV-O
- NIST SP 800-53A Rev. 5
- NIST AI 600-1: Generative AI Profile
Sources: [1] [2] [3]
Sources
- 1.W3C PROV-O · World Wide Web Consortium
- 2.NIST SP 800-53A Rev. 5 · National Institute of Standards and Technology
- 3.NIST Generative AI Profile · National Institute of Standards and Technology