How to develop and assess relevance judgment in AI-enabled work

Define relevance as a relationship, not a property

The U.S. Government Accountability Office's 2024 Government Auditing Standards treats relevance as one aspect of appropriate evidence and describes it in terms of the logical relationship to, and importance for, the matter being addressed. NIST's AI Risk Management Framework Core likewise asks teams to establish context, intended purpose, scope, limitations, and the relevance of knowledge to that context.

These principles point to a practical rule: evidence is relevant to a specified proposition in a specified context. There is no universal relevance score attached to a document. Change the decision, time period, population, or level of detail, and the same document may move from essential to incidental.

Build a BEARS map

Use a BEARS map before writing. Each row connects one evidence item to one decision-bearing proposition.

| Field | Test | What to record | | --- | --- | --- | | Brief | What exact decision or question must the work product support? | One sentence with the decision owner and deadline | | Evidence proposition | What specific statement could this item support or challenge? | A bounded claim, not a topic label | | Applicability | Does the item cover the right population, process, geography, and conditions? | Matches and mismatches | | Recency | Is its observation period suitable for this decision? | Relevant dates and any staleness risk | | Significance | If the item changed, could the recommendation or confidence change? | Decision impact: high, medium, low, or none |

The map prevents two common errors. The first is topical substitution: including a document because it mentions the same subject. The second is prestige substitution: including it because its publisher is respected. Neither establishes that the information answers the question.

Start with the brief, not the source pack. Rewrite a broad assignment into one decision sentence and two to four propositions that must be resolved. Then examine each source for the particular proposition it can support, contradict, or leave uncertain. If no proposition changes, the item belongs in background reading or outside the work product.

Make exclusion part of the work

A good relevance decision leaves a trace of what was excluded and why. Add an exclusion column to the BEARS map and use concrete reasons such as "wrong reporting period," "different customer segment," "describes adoption but not outcome," or "duplicates the underlying dataset." Avoid vague labels such as "not useful."

AI can help propose candidate links between evidence and propositions. It can also challenge a map by asking what would change if a source were removed. The worker should still inspect the source and decide whether the asserted relationship is real. A generated explanation of relevance is not evidence of relevance.

Use a counterfactual check for high-impact items: if this source's key fact were reversed, would the recommendation, option ranking, or confidence change? If nothing would change, either the item is contextual rather than decision-bearing, or the decision logic has not been made explicit.

Worked example: selecting evidence for an operations decision

Suppose a programme lead must decide whether to extend a synthetic internal pilot for another six weeks. This example is invented for illustration and does not report a Two Prune customer result.

The evidence pack contains a current completion log, participant comments, a two-year-old industry survey about AI interest, an internal support-ticket export, and a vendor article about general productivity. The initial AI summary cites all five sources because each mentions AI adoption or programme experience.

The lead turns the brief into three propositions:

The completion log has high relevance to the first proposition when its denominator, dates, and definition of completion match the pilot. The support-ticket export may bear on the second proposition if tickets concern the current workflow and show a recurring, remediable issue. Participant comments provide context and candidate explanations, but a few comments cannot establish prevalence.

The industry survey describes a different population and predates the pilot. It may explain the wider environment but cannot answer whether this programme should continue now. The vendor article is also background: it discusses general productivity and contains no evidence about this pilot, its participants, or its operational constraint.

The resulting one-page map keeps the completion log and ticket export in the decision path, labels the comments as exploratory context, and excludes the other two sources from the recommendation. The lead proposes a six-week extension with a defined completion threshold and a review of the identified support issue. The example does not claim the extension will work. It shows how relevance narrows the evidence to the actual choice.

• Enough participants are completing the current pilot to justify another cycle.
• A recurring operational problem can be addressed within the proposed extension.
• The extension has an observable completion condition rather than an open-ended promise.

Practice with decision changes

Relevance becomes easier to see when the same pack is used for different briefs. Give a learner one small evidence pack, then change the question three times. For example: decide whether to extend the pilot; explain why completion varied; design a communication plan. Ask the learner to rebuild the BEARS map for each question.

The selected evidence should change. A participation log may be central to the extension decision, one input among several for causal exploration, and only a segmentation aid for communication planning. This exercise discourages a static "good sources" list and develops sensitivity to purpose, scope, and consequence.

Another useful drill is the removal test. Delete one source from a draft and ask the learner to identify exactly which sentence, inference, or confidence judgment must change. If the answer is "none," challenge whether the source was decision-bearing at all.

Assess observable judgment

An assessment can present a realistic workplace brief and a mixed pack containing current records, loosely related commentary, a prestigious but out-of-scope publication, and one evidence gap. Ask the participant to deliver a recommendation plus the BEARS map used to select evidence.

Review these observable behaviors:

Do not score relevance by the number of sources cited or by agreement with a preferred recommendation. A shorter evidence set can show stronger judgment when exclusions are well reasoned. Retain the source-selection map, exclusion notes, queries, opened-source history, and revisions as evidence of the process.

• the decision question is specific enough to test relevance;
• evidence is mapped to bounded propositions rather than broad topics;
• applicability and dates are checked instead of assumed;
• material evidence is distinguished from useful context;
• exclusions have precise, defensible reasons;
• missing relevant evidence is named rather than filled with generated certainty;
• the final recommendation changes appropriately when a decisive item changes.

Verify the final evidence path

Read the work product backward from its decision. For each recommendation or material conclusion, identify the proposition it depends on and the evidence selected for that proposition. Then check the evidence's population, period, conditions, and level of detail against the brief. Finally, inspect excluded material to make sure a disconfirming item was not dismissed merely because it was inconvenient.

Relevance judgment improves the fit between evidence and a decision. It does not establish source authority, prove causation, or ensure that the decision is correct. Within a Two Prune simulation, the map and work trail can contribute evidence about Evidence Navigation, Context Framing, and Judgment & Synthesis. They produce a bounded assessment signal about observed performance, not a prediction of future results or an automated employment decision.

Sources

U.S. Government Accountability Office, Government Auditing Standards 2024 Revision

• NIST AI Risk Management Framework Core

Sources: [1] [2]