How to develop and assess fact checking in AI-enabled work

Turn prose into a claim inventory

The Government of Canada's guide on generative AI tells public servants to evaluate outputs for factual inaccuracies and to record accuracy review where appropriate. Its content guidance for AI help applications says citations should direct people to authoritative source material they can inspect. The NIST Generative AI Profile recommends reviewing sources and citations in generated content during pre-deployment testing. These are useful practices, but no source list can remove the need for judgment about scope and support.

Begin by underlining statements that could be true or false. Split compound sentences when their parts need different evidence. Give each claim a type:

The type affects the check. A current status needs a current owner or system of record. A historical quantity needs the edition and measurement definition. A causal claim needs more than two facts that occurred together. A prediction must remain labelled as a forecast and show its assumptions.

Also record materiality. A spelling error in a background detail and an incorrect eligibility rule should not receive the same review effort. Materiality depends on what the reader may decide, what can be reversed, who may be affected, and how quickly a correction could reach them.

• identity: who or what an entity is;
• quantity: a number, proportion, price, date, duration, or threshold;
• status: whether something is active, approved, required, available, or complete;
• relationship: whether two events or variables are associated;
• causal: whether one thing produced a result;
• comparison: whether one option is larger, faster, cheaper, or otherwise different;
• prediction: what is expected to happen;
• quotation or attribution: what a named source said or did.

Build a FACT docket

Use one FACT docket per material claim:

| Field | Question | Evidence retained | | --- | --- | --- | | Form | What is the smallest exact claim, and what type is it? | Quoted sentence, location, and claim type | | Authority | Which source is closest to the fact and responsible for maintaining it? | Primary record, owner, edition, date, and access path | | Context | Which population, period, definition, exception, and comparison frame apply? | Relevant passage plus excluded conditions | | Test | Does the evidence support, narrow, contradict, or leave the claim unresolved? | Reasoning, corroboration where useful, and disposition |

The source closest to the fact is not always a webpage that repeats it. For a filing date, use the filing or responsible register. For a product configuration, use the actual approved setting or current first-party documentation. For survey results, use the report and methodology, not a reposted chart. For a policy status, use the policy owner and effective version.

Corroboration has a purpose. A second site that copies the first is not independent support. Look for a separately maintained record, a direct observation, or a source with a different evidence path. If the primary owner is unavailable or disputed, record that limitation rather than promoting repetition to proof.

Worked artifact: checking a readiness update

Consider an original synthetic example. A programme lead asks an AI assistant to draft an internal update from workshop notes, a survey export, a project tracker, and a vendor announcement. The draft contains three material claims:

The checker creates three FACT dockets.

### Claim 1: readiness

Form: quantity and status claim. The exact wording says that 84% of all employees are ready.

Authority: the survey export contains 84 positive answers among 100 respondents. The workforce count is 640, and participation was voluntary.

Context: the question asked, “How confident are you that you could try the assistant in a guided exercise?” It did not define readiness and did not test performance.

Test: the arithmetic describes respondents, not employees, and the question measures self-reported confidence about trying a guided exercise. The disposition is replace:

> In the voluntary pulse survey, 84 of 100 respondents said they were confident they could try the assistant in a guided exercise.

The update adds the response scope and avoids treating confidence as observed capability.

### Claim 2: security review

Form: current status claim.

Authority: the project tracker marks the review “complete” on 12 September, but the attached review note has two open conditions and says production access remains pending.

Context: the tracker field represents completion of the review meeting, not closure of every condition.

Test: the original wording would invite readers to infer approval. The disposition is narrow:

> The security review meeting took place on 12 September. Two conditions remain open, and production access has not been approved.

### Claim 3: handling time

Form: causal comparison.

Authority: workshop notes compare the median handling time for 20 assisted practice cases with 20 earlier cases completed by a different group. The cases were not matched for difficulty, and the tracker contains no quality measure.

Context: the exercise was synthetic, the groups differed, and the records do not isolate the effect of the assistant.

Test: the observed medians may be reported, but the verb “cut” makes a causal claim that the evidence does not establish. The disposition is replace:

> In this synthetic exercise, the assisted group's median recorded handling time was 30% lower than the earlier comparison group's median. The groups and cases were not matched, so the difference should not be attributed to the assistant.

These values and records are invented. They are not customer outcomes, benchmarks, or claims about Two Prune. The artifact demonstrates how one polished update can contain three different evidence problems that require different corrections.

• “Eighty-four percent of employees are ready to use the new assistant.”
• “The security review was completed on 12 September.”
• “The pilot cut handling time by 30 percent.”

Practise with a fact-checking ladder

Give a learner a one-page AI-generated brief and a small source pack. Ask them to complete three rounds:

Then introduce a source update that changes one date or definition. The learner should locate every affected claim. This tests whether fact checking is connected to the document rather than performed as a detached checklist.

AI may help extract possible claims or compare passages. The learner must verify the extraction, choose authoritative records, inspect source context, and decide what wording the evidence permits. A generated explanation of why a claim is true is not evidence for the claim.

• Inventory: extract every material checkable claim without opening the sources.
• Resolution: complete FACT dockets using the source pack and identify what cannot be resolved.
• Propagation: revise the brief, its title, table, summary, and recommendation so every dependent statement reflects the dispositions.

Keep fact checking distinct from adjacent controls

Several practices overlap, but their outputs differ:

A fact check may invoke the other controls. For example, a quantity docket may require a reconstructed calculation, and a quotation docket may require citation verification. The FACT docket remains the claim-level record that connects those checks to the final wording.

• citation verification asks whether a cited object is authentic, current, correctly identified, and actually supports the nearby statement;
• calculation checking reconstructs numerical operations from inputs, rules, units, and denominators;
• hallucination detection looks for content that appears invented or unsupported by the provided evidence;
• fact checking assigns a disposition to a precise claim after testing authority, context, and support.

Assess fact checking from observable work

Review the claim inventory, dockets, source passages, revisions, and final communication:

Do not score only the number of errors found. A learner who marks many harmless details but misses one decision-changing status claim has not demonstrated strong prioritisation. Conversely, a learner may appropriately retain a claim after documenting why its evidence and scope are sufficient.

• coverage: did the learner identify the material claims a reader might rely on?
• precision: did they reduce broad or compound prose to exact testable statements?
• authority: did they use the source closest to the fact and explain any substitution?
• context control: did they preserve dates, populations, definitions, exceptions, and measurement conditions?
• claim logic: did they distinguish observation, association, causation, forecast, and opinion?
• disposition quality: did they retain, narrow, replace, remove, or hold each claim based on evidence?
• propagation: did corrections reach summaries, tables, headings, recommendations, and downstream calculations?
• uncertainty: did unresolved claims remain visibly unresolved rather than disappear into confident prose?

Close with a release check

Before release, compare the final wording to every FACT disposition. Confirm that the linked source still resolves, that its edition and effective date match the claim, and that quotations or extracted values have not drifted during editing. Search the document for old numbers, dates, labels, and causal verbs that may survive outside the corrected paragraph.

For a time-sensitive claim, name the “as of” date and assign an owner or refresh condition. For a consequential unresolved claim, remove it from decision-facing prose or escalate it with the exact evidence gap. Keep the docket with the version reviewed so that a later correction can be traced.

Fact checking reduces the risk of releasing unsupported statements. It does not prove that the source itself is error-free or that every relevant fact was available. The Government of Canada's guidance pairs factual review with attention to bias, harm, transparency, and documentation. NIST likewise places source and citation review among a broader set of generative-AI testing and risk-management actions.

In a Two Prune assessment, the claim inventory, FACT dockets, and propagated revisions can provide task-bounded evidence about Verification & Risk Control, Evidence Navigation, and Communication & Delivery. They do not certify a participant as a professional fact checker, predict workplace outcomes, or make an employment decision.

Sources

Government of Canada: Guide on the use of generative artificial intelligence

• Government of Canada: Content guidance for AI help applications
• NIST: Artificial Intelligence Risk Management Framework, Generative Artificial Intelligence Profile

Sources: [1] [2] [3]