How to develop and assess evidence weighting in AI-enabled work

Weight evidence for a claim, not for a source in the abstract

The same source can be strong for one claim and weak for another. A current operational log may directly establish how many cases entered a workflow, yet say little about why completion changed. A controlled study may support a general mechanism but not match the population, setting, or implementation under review. Evidence weighting therefore begins with a bounded question and a set of claim-sized propositions.

The UK National Screening Committee's evidence review process considers the quantity of evidence, its quality or risk of bias, applicability, and consistency when deciding whether criteria are met. The World Health Organization describes evidence-informed decision-making as identifying, appraising, and mobilizing the best available evidence. These principles are useful beyond health policy: evidence has to be appraised and used in context, not counted as votes.

Avoid assigning one permanent “trust score” to a publisher or document. Authority matters, but it does not repair a mismatch between the evidence and the claim. Likewise, five reports repeating one upstream dataset are not five independent confirmations. Trace dependence before adding apparent support.

Use a TRACE ledger

Create one row per evidence item and claim. Use TRACE to record why an item should influence the conclusion:

| Field | Test | Possible record | | --- | --- | --- | | Target | Which exact proposition does this evidence address? | Claim ID and relevant passage or field | | Reliability | How was the record produced, checked, and exposed to error? | Method, missingness, risk notes, version | | Applicability | How directly does the population, setting, time, and intervention match? | Match and mismatch statement | | Connection | Is this independent evidence or derived from another item? | Provenance link and dependency cluster | | Extent | How precise, complete, and consistent is the support? | Range, sample coverage, conflicts, uncertainty |

TRACE is not a mechanical points system. Its purpose is to force a reasoned judgment at the claim level. For each row, assign an influence category such as decisive, supporting, contextual, or excluded, followed by one sentence explaining the category. Reserve “excluded” for evidence that cannot answer the proposition or has a material defect, and retain the exclusion reason so inconvenient evidence does not silently disappear.

Then make a claim summary. State the direction of the combined evidence, the main strength, the main limitation, and the conclusion that the current evidence permits. If important items conflict, describe the most plausible reason for the conflict and what evidence could distinguish among explanations.

Worked artifact: conflicting completion records

Imagine a synthetic learning team evaluating whether to continue a new guided practice format. This example is original and does not report a Two Prune result, customer outcome, or established benchmark.

The decision claim is narrow: “During the eight-week pilot, the guided format was associated with higher completion of the assigned practice sequence than the previous self-directed format.” The evidence pack contains four items:

The analyst first checks definitions. The current export defines completion as submitting all four tasks. The slide counts anyone who opened task four, so it does not measure the target. The survey asks whether the format helped respondents stay engaged, which is relevant context but cannot establish completion for non-respondents. The prior-cycle export uses the same four-submission definition after a data dictionary is recovered.

The TRACE ledger assigns influence as follows:

After recalculation, 55 of 84 pilot participants submitted all four tasks, compared with 43 of 79 in the prior cycle. Excluding the seven late enrolments changes the pilot figure to 54 of 77. The direction remains the same, but the analyst does not convert the difference into a causal claim. Other changes between cycles may explain some or all of it.

The decision note says the operational records support continuing the format for one more cycle while standardizing exposure dates and collecting a comparable reason-for-noncompletion field. It labels the survey themes as hypotheses for that follow-up. The artifact includes the TRACE ledger, both calculations, the definition crosswalk, and a list of unresolved alternative explanations.

• a current activity export covering all 84 enrolled participants in the pilot;
• an administrator slide saying completion was 72 percent;
• a voluntary feedback survey answered by 31 participants;
• a prior-cycle activity export for 79 participants using the older format.
• Current activity export: decisive for the pilot completion rate. It directly covers the target population and definition. Seven late enrolments had less than eight weeks of exposure, so the analyst reports results both with and without them.
• Administrator slide: excluded from the numerical comparison. Its numerator uses a different event. The slide is retained as evidence of the definition mismatch, not silently discarded.
• Feedback survey: contextual. It may help generate explanations, but self-selection and the smaller respondent group limit its use for the completion comparison.
• Prior-cycle export: supporting for the historical comparison. It is direct and definition-matched, but the cohorts occurred at different times and were not assigned randomly, so the contrast does not establish that the format caused the difference.

Practice by changing one evidence property at a time

Evidence weighting improves when learners can see how a conclusion should move. Give them a stable claim and several short records. Then alter one property:

Ask the learner to revise only the affected TRACE entries and conclusion. The task is not to reward a particular hierarchy of source types. It is to observe whether the person can explain why influence changed.

AI can help build the ledger, compare definitions, surface possible dependencies, and draft alternative interpretations. Require the learner to open the underlying records, verify the relevant passages or fields, and correct the model when it treats citation frequency as corroboration. A useful prompt is: “What would make this item less influential for this exact claim?” The answer should lead to a check, not merely a confidence adjective.

• make a source more direct but less complete;
• reveal that two reports share the same underlying dataset;
• change the population so an otherwise strong study is less applicable;
• add a precise measurement that conflicts with a broad self-report;
• update a record after the initial recommendation has been drafted.

Assess the reasoning and retained evidence

Use observable criteria rather than judging whether the prose sounds cautious:

Retain the bounded claim, TRACE ledger, source locations, dependency map, calculations, excluded-item reasons, sensitivity result, and revision trail. A long bibliography or a model-produced confidence score does not demonstrate weighting skill.

• claim targeting: evidence is mapped to propositions narrow enough to test;
• method inspection: production method, missingness, version, and material error risks are checked;
• applicability judgment: population, setting, timing, intervention, and outcome mismatches are explicit;
• dependency control: repeated use of the same underlying evidence is not mistaken for confirmation;
• conflict handling: contrary evidence is investigated, retained, and explained;
• proportional conclusion: the recommendation does not exceed what the weighted evidence permits;
• sensitivity: removing or downgrading a pivotal item triggers a documented re-evaluation;
• correction behaviour: discovered definition or calculation problems change the ledger and conclusion.

Verify the weighting before delivery

Perform a leave-one-out test on every decisive item: remove it temporarily and state how the conclusion changes. If nothing changes, the item may not be decisive. If the recommendation collapses, make that dependency visible. Next, group items by upstream origin and repeat the test at the cluster level. This catches false corroboration.

Check every conclusion verb against the evidence. “Recorded,” “reported,” “associated,” and “caused” require different support. Recalculate material figures, reconcile definitions, and identify the oldest evidence that still influences the result. Finally, ask a reviewer to argue the strongest contrary interpretation using the retained pack.

Evidence weighting makes the basis and limits of a judgment visible. It does not turn heterogeneous evidence into objective certainty or remove the need for accountable human interpretation. In a Two Prune assessment, the artifact can support observations about Evidence Navigation, Judgment & Synthesis, and Verification & Risk Control in the assigned simulation. It is an assessment signal about demonstrated work, not an automated employment decision.

Sources

UK National Screening Committee: evidence review process

• World Health Organization: Evidence, policy, impact
• World Health Organization: Evidence-to-Decision tables and framework

Sources: [1] [2] [3]