Practical guide
What evidence shows model selection is operating?
Model selection is operating when an organization can reconstruct why a particular method was chosen for a defined use, which alternatives were considered, how evidence matched the intended conditions, who challenged the choice, what limits were approved, and whether later changes still fit that decision. A model inventory entry or vendor comparison alone is not evidence that selection worked.
Model selection is operating when an organization can reconstruct why a particular method was chosen for a defined use, which alternatives were considered, how evidence matched the intended conditions, who challenged the choice, what limits were approved, and whether later changes still fit that decision. A model inventory entry or vendor comparison alone is not evidence that selection worked.
Define the control objective
The objective is: each material model or adjacent AI use has a documented selection decision that compares credible alternatives against pre-defined requirements, uses reproducible evidence, records independent challenge and authority, and remains linked to the deployed version and approved use.
Federal Reserve SR 26-2 provides current supervisory guidance for model risk management at specified banking organizations and expressly excludes generative and agentic AI from that guidance. NIST’s voluntary AI Risk Management Framework Core addresses inventories, roles, requirements, measurement, monitoring, and treatment more broadly. Use these sources within their stated scopes; route legal, regulatory, safety, and sector conclusions to qualified reviewers. Federal Reserve: SR 26-2 NIST: AI RMF Core
Sources: [1] [2]
Test selection evidence with SELECT
### S — Scope and selection question
The record states the business purpose, decision consequence, users, affected groups, input data, output use, fallback, materiality, risk tier, constraints, and prohibited uses. It identifies the decision owner and approval date.
Test whether the question was specific enough to compare methods. “Choose the best AI model” is not testable. “Select a method for routing approved internal service requests into 12 categories, with human review and escalation for ambiguous cases” is.
### E — Eligible alternatives and baseline
Require a credible baseline and materially different alternatives. These may include no automation, process redesign, rules, statistical methods, machine-learning models, a vendor service, or a human-supported workflow. Record exclusion reasons before final results where possible.
Test for a predetermined winner: different requirements, data, populations, time windows, or cost assumptions applied to one alternative are a failure. A preferred vendor’s availability is a constraint, not proof of suitability.
### L — Lineage of data, versions, and evaluation
The evidence connects source data, permissions, sampling, labels, exclusions, transformations, versions, configuration, environment, prompts or features where relevant, evaluation code, outputs, and review artifacts. It states which conditions resemble production and which do not.
Reproduce a sample from source through result. Confirm that the evaluated version is the one proposed for release. Screenshots, summary slides, and vendor reports can supplement but not replace controlled evaluation evidence.
### E — Evaluation across conditions and consequences
Measures must follow from the use and consequence. Evaluate relevant performance, error types, calibration or confidence behavior, stability, boundary cases, missing inputs, subgroups where lawful and meaningful, operational burden, human review, fallback, and failure visibility.
Use distributions and scenarios rather than one average when errors have different consequences. Record uncertainty and sensitivity. A higher headline metric does not win automatically if it creates an unacceptable failure mode or cannot operate inside required controls.
### C — Challenge, contradictions, and competence
Independent challenge tests conceptual fit, data suitability, alternative coverage, assumptions, evaluation design, implementation, limitations, and evidence gaps. The challenger has sufficient access, competence, authority to disagree, and freedom from being the sole author of the selected method.
Preserve dissent and contradictory results. Record reviewer preparation, conflicts, findings, dispositions, and closure evidence. An unsigned “reviewed” status is not challenge evidence.
### T — Treatment, approval, and continued fit
The decision records select, limited test, repair, hold, or reject; the exact version; approved and prohibited uses; users; human review; monitoring; residual uncertainty; and review date. It links to release configuration and change triggers.
Continued-fit evidence checks whether data, method, vendor version, integration, population, use, or consequences changed. A selection can expire even when the system still runs.
Minimum evidence ledger
| Evidence | Minimum content | Operating test | | --- | --- | --- | | Selection brief | purpose, consequence, scope, owner, requirements | question is specific and authorized | | Alternatives register | baseline, candidates, exclusion reasons | alternatives are credible and comparable | | Data and version manifest | sources, population, transformations, versions | lineage reproduces | | Evaluation plan | measures, scenarios, thresholds, uncertainty | tests match consequence | | Results packet | outputs, errors, sensitivity, burden | calculations reproduce | | Challenge record | reviewer, competence, findings, dissent | issues have dispositions | | Selection decision | exact version, use, limits, residual risk | approval matches evidence | | Release match | deployed configuration and controls | production matches decision | | Continued-fit review | changes, monitoring, incidents, date | selection remains applicable |
Sample approvals, limited tests, holds, and rejections. Testing only successful selections can hide whether the control detects weak evidence.
Worked evidence test: service-request routing
A synthetic operations team selects between keyword rules, a supervised classifier, and a general-purpose language model for routing internal service requests into 12 categories. A slide says the language model is “most accurate” at 91% and recommends production use.
The Scope test finds that the decision actually requires safe escalation for ambiguous access incidents, not just overall accuracy. The baseline human-assisted workflow and improved request form were omitted. They are added to the alternatives register.
Lineage review finds that the language-model evaluation used a newer request sample than the classifier. It also included repeated requests from the same incident in both development and evaluation sets. The team rebuilds a time-separated, deduplicated sample and freezes all versions.
Condition testing reports per-category errors, ambiguous cases, missing context, confidence behavior, reviewer correction time, and fallback success. The language model still has the highest average routing accuracy, but it invents unsupported rationale and sends more access incidents to account provisioning. The classifier is less accurate overall yet exposes stable confidence and routes uncertain cases to review.
Independent challenge questions whether the labels themselves match the current service taxonomy. A source review confirms two overlapping categories. The decision owner cannot distinguish method error from taxonomy ambiguity, so the selection disposition is repair and limited comparison, not approval.
The team fixes the taxonomy, adds a frozen source date, tests rules plus classifier as a combined alternative, and defines an escalation threshold. The language model remains in shadow evaluation. The evidence packet includes the revised brief, alternatives, dataset manifest, deduplication test, evaluation code, error analysis, correction burden, challenger dissent, repair decision, and next review date.
This result does not prove that the classifier is safe, compliant, or generally superior. It shows that the organization’s selection evidence detected an invalid comparison and routed the decision to repair.
Failure and hold conditions
Hold selection when the use or consequence is unclear, the baseline is missing, alternatives receive different data or criteria, lineage cannot reproduce, the evaluated and proposed versions differ, decisive error types are hidden by an average, challenge lacks competence or independence, findings lack dispositions, or approval exceeds tested conditions.
Treat policy design, sampled control operation, and supported effect as separate claims. A complete evidence packet can show that required steps occurred in the sampled decision; it does not alone prove organization-wide control effectiveness.
Two Prune can place model-selection evidence review inside a realistic AI-enabled assessment and evaluate how a participant defines requirements, navigates sources, compares alternatives, challenges results, and communicates a defensible decision. It does not select production models, certify controls, or make employment decisions.
Sources
Board of Governors of the Federal Reserve System: SR 26-2 revised guidance on model risk management
- National Institute of Standards and Technology: AI Risk Management Framework Core
Sources: [1] [2]
Sources
- 1.Federal Reserve: SR 26-2 · www.federalreserve.gov
- 2.NIST: AI RMF Core · National Institute of Standards and Technology