How to curate an evidence collection with AI

Write the collection charter

State the decision or research question, audience, jurisdictions, time period, source types, authority hierarchy, inclusion and exclusion rules, update cadence, and stopping condition. Define whether the collection is comprehensive, representative, or intentionally selective.

The U.S. National Archives maintains metadata requirements for permanent electronic records and brings together requirements for descriptive and transfer metadata. Its guidance is specific to federal records, but it illustrates why context and provenance fields must travel with an item.

Build a source hierarchy

Classify sources before collection. A common hierarchy might separate enacted instruments and standards, issuing-body guidance, primary research or datasets, organizational records, expert analysis, and discovery-only material. Relevance and authority are separate: a highly authoritative source may not answer the question, while a relevant commentary may not establish the underlying fact.

Worked synthetic collection

Suppose a team is assembling evidence for the question: “What controls should govern external generative-AI tools used with confidential assessment materials?”

Its manifest might include:

| ID | Item | Authority | Date/version | Inclusion reason | Limitation | | --- | --- | --- | --- | --- | --- | | S1 | Internal data-classification policy | Governing internal policy | Current signed version | Defines confidential material | Does not address AI tools | | S2 | Security architecture standard | Approved internal standard | v4 | Defines approved processing patterns | Pending next review | | S3 | Vendor data-flow documentation | Supplier first-party | Retrieved date | Describes claimed processing | Requires technical verification | | S4 | Regulator guidance | Issuing authority | Current page | Provides relevant risk framing | Applicability needs qualified review |

The example does not determine which control is required.

Capture item-level provenance

Record stable identifier, title, creator, publisher, URL or repository path, publication and retrieval dates, version, jurisdiction, license or access condition, checksum where appropriate, and exact passages used. Keep the original file and a normalized working copy distinct.

Ask AI to propose metadata from the document, then verify it against the source page or record. If author, date, or status is absent, store unknown rather than infer it.

Apply inclusion rules consistently

Screen every candidate against the charter and record a disposition: include, exclude, replace with authoritative version, or hold for review. Use specific reasons such as “superseded by S2 v4” or “describes public data only.” Retain the decision log so later users can see what was considered.

Test collection coverage

Create a question-to-source matrix. Mark which source supports each component and where conflicts remain. Search for missing perspectives, dates, jurisdictions, adverse findings, and implementation evidence. More URLs do not repair an empty evidence cell.

For the synthetic question, the matrix may reveal strong policy and vendor documentation but no tested evidence of the actual data flow. That is a verification gap, not permission to repeat the vendor statement as fact.

Produce the collection package

Deliver the charter, manifest, source files or controlled links, passage notes, inclusion log, conflict register, coverage matrix, and update schedule. Explain access restrictions and handling requirements.

Verification checklist

Confirm every item meets the charter; provenance fields are source-verified; versions and dates are visible; duplicates and superseded items are controlled; conflicts remain explicit; evidence passages support their labels; permissions are respected; gaps are documented; and updates have an owner.

Sources and claim map

The charter, sources, and control question are synthetic. This educational resource does not establish legal applicability, security sufficiency, or completeness of any real evidence collection.

• U.S. National Archives, Metadata Requirements for Permanent Electronic Records: supports preserving defined metadata with official records in its federal scope.
• U.S. National Archives, Records Management Profile: supports retaining business context and record-level metadata across a records lifecycle.

Sources: [1] [2]