Checking selection bias in AI-assisted analysis

Draw the entry path before interpreting a pattern

Imagine a synthetic support team asking an AI assistant to summarize “what customers thought of the new onboarding.” The input contains 180 tickets from a help inbox. The assistant reports that most customers found setup confusing. But the help inbox contains people who had a reason to contact support; customers who completed setup without asking for help do not appear there. The summary may accurately describe the inbox and still be an invalid statement about all customers.

The analyst's sample-entry ledger begins with the intended decision: should the team revise onboarding for the full user base? It then lists the target group, the observed frame, the selection mechanism, and missing groups. Target: everyone who tried the new setup during the month. Observed frame: tickets assigned an onboarding label in the help inbox. Entry mechanism: a person had to encounter a problem, choose to contact support, and receive the label. Missing or weakly represented groups include people who completed setup silently, abandoned without writing, or contacted another channel.

The GAO AI Accountability Framework identifies data quality, reliability, and representativeness as accountability concerns. It speaks to AI systems in a broad organizational context, not specifically to this fictional inbox. The NIST AI Risk Management Framework Core likewise calls for context-appropriate evaluation, attention to deployment conditions, and documented limitations. These sources support checking representativeness; they do not supply a numeric correction for this example.

Sources: [1] [2]

Test what the sample excludes

Count tickets by channel, period, label, product version, and customer segment before accepting the assistant's themes. Compare those counts with an authorized source of all onboarding attempts if one exists. If 180 tickets came from 4,000 attempts, that ratio is not a satisfaction score: many non-ticket users may have had trouble, and some tickets may concern unrelated issues. Inspect a small sample of excluded records, including tickets without the onboarding label, to see whether the filter omitted relevant cases. Preserve the query and version used to assemble the dataset.

In the synthetic case, the team discovers that a migration changed ticket labels mid-month. The assistant's input includes nearly all week-one tickets but only some week-four tickets. A second extract using the revised label finds additional cases. The comparison does not make the inbox representative of all customers; it merely corrects one avoidable omission. The analysis should now be labeled “themes among support contacts under two ticket-label versions.”

The ledger also records who was able to be observed and who was not. If some customers used a reseller or an in-product chat whose records were inaccessible, note that boundary. Do not ask the AI assistant to invent what those groups thought. A source that cannot be obtained is a limitation to communicate, not a reason to silently treat the available frame as complete.

Release a conclusion matched to the evidence

A defensible support-team note might say: “In reviewed onboarding-related support contacts, account permissions and the first data import were common sources of confusion. We have not measured prevalence among all new users.” That is actionable: the team can test those steps with a broader sample or a targeted user study. It avoids claiming that most customers struggled merely because most selected tickets mention a problem.

Ask a reviewer who did not assemble the extract to reconstruct the entry filter, reproduce counts for one week, and identify the largest missing group. Compare the final headline with the ledger's target population. If the headline speaks about all customers while the evidence covers only support contacts, revise it before release. The synthetic numbers are teaching examples, not customer results or a benchmark.

Two Prune can observe whether a participant notices an evidence-selection problem inside a bounded assessment and report available task evidence for human interpretation. It does not measure every real-world customer population, certify fairness, or issue automated employment decisions.