Practical guide
Model-selection policy for AI-enabled work
A model-selection policy should require every material choice of a model or adjacent AI method to begin with a defined use, credible alternatives, comparable evidence, independent challenge, authorized approval, and continued-fit review. It should prevent a team from treating vendor availability, a single benchmark, or deployment success as proof that a method is appropriate.
A model-selection policy should require every material choice of a model or adjacent AI method to begin with a defined use, credible alternatives, comparable evidence, independent challenge, authorized approval, and continued-fit review. It should prevent a team from treating vendor availability, a single benchmark, or deployment success as proof that a method is appropriate.
Source and scope boundary
This is a policy-design guide, not jurisdiction-specific legal advice. Federal Reserve SR 26-2 applies to specified banking organizations and expressly excludes generative and agentic AI. The NIST AI Risk Management Framework is voluntary and broader. Organizations should define applicable obligations and route systems outside a model definition into another accountable governance path. Federal Reserve: SR 26-2 NIST: AI RMF Core
Sources: [1] [2]
Policy objective and scope
The objective is: no material method is selected, released, or materially changed without a documented use, selection criteria, credible baseline and alternatives, controlled evaluation, independent challenge, approved limits, release match, and review date.
Define model, AI system, material use, selection, baseline, alternative, version, independent challenge, approval, exception, change, and retirement. State covered business units, systems, vendors, spreadsheets, rules, statistical methods, machine-learning models, generative AI, temporary tools, and development environments. Where a category is excluded, name its governance route and owner.
Use the CHOICE policy structure
### C — Context, consequence, and classification
Every selection begins with purpose, decision consequence, users, affected groups, data, output use, fallback, materiality, risk tier, constraints, and prohibited uses. The control owner records whether the system is in scope and which reviewers are required.
Unresolved classification is a hold. The policy should not allow teams to avoid requirements by calling a production method a prototype.
### H — Head-to-head alternatives and baseline
Require a credible no-change or current-process baseline plus materially different alternatives. Alternatives may include process redesign, rules, human-supported workflows, statistical methods, models, or vendor services.
Selection criteria must be defined before final comparison and applied consistently. Mandatory legal, security, privacy, accessibility, safety, data, or operational gates cannot be offset by a high average score.
### O — Objective evidence and reproducible evaluation
Evaluation records include data authority, sampling, labels, exclusions, transformations, versions, configuration, environment, measures, thresholds, scenarios, uncertainty, human review, operational burden, and limitations. The evaluated and proposed versions must match.
Measures follow the intended consequence. Require error types, boundary cases, missing inputs, relevant populations, fallback, failure visibility, sensitivity, and uncertainty where appropriate. A benchmark that does not represent the approved use is context, not acceptance evidence.
### I — Independent challenge and issue disposition
The challenger must have access, competence, authority to disagree, and freedom from being the sole author of the selected method. Challenge covers conceptual fit, alternative coverage, data suitability, evaluation design, implementation, limitations, and contradictions.
Every material finding records severity, impact, owner, disposition, due date, and closure test. Preserve dissent. A status of “reviewed” without findings and evidence is insufficient.
### C — Conditions, approval, and change control
The approver selects approve, limited test, repair, hold, or reject and records the exact version, users, approved and prohibited uses, human review, monitoring, residual uncertainty, and review date.
Release control verifies production configuration and controls against the decision. Changes to data, method, vendor, version, integration, population, use, consequences, or jurisdiction enter triage before implementation. The control owner decides whether to update evidence, rechallenge, reapprove, restrict, or stop.
### E — Exceptions, evidence retention, and expiry
Exceptions identify exact scope, reason, authority, compensating controls, evidence, expiry, and stop trigger. They cannot permanently expand an approval by repeated renewal.
Retain the selection brief, alternatives register, data/version manifest, evaluation plan and results, challenge, dissent, decision, release match, monitoring, changes, issues, and continued-fit review. Set review cadence and expiry based on risk and change.
Minimum policy record
| Record | Required content | | --- | --- | | Selection brief | use, consequence, scope, owner, requirements, risk tier | | Alternatives | baseline, candidates, exclusions, common criteria | | Evidence manifest | data, versions, configuration, environment, lineage | | Evaluation | measures, scenarios, thresholds, results, uncertainty, burden | | Challenge | reviewer competence, tests, findings, dissent, dispositions | | Decision | exact version, use, conditions, residual risk, approver, date | | Release match | deployed configuration, access, controls, communication | | Continued fit | changes, monitoring, incidents, exceptions, next review |
Operating procedure
Requester submits the use, consequence, users, data, fallback, and owner.
- Control owner classifies scope and risk tier.
- Owner defines requirements, baseline, alternatives, and evidence plan.
- Technical and data owners produce controlled evaluation evidence.
- Independent challenger tests fit, methods, results, and limitations.
- Findings receive documented dispositions and closure tests.
- Authorized approver issues approve, limit, repair, hold, or reject.
- Release owner verifies production matches the approved version and controls.
- Operational owner monitors conditions and routes breaches or changes.
- Control owner reviews continued fit, exceptions, and expiry.
Worked policy test: internal request routing
A synthetic team proposes a general-purpose language model for routing internal service requests. A slide reports 91% accuracy and says it outperforms a supervised classifier.
CHOICE requires the current human-assisted process and improved keyword rules as alternatives. The data manifest reveals that the language model used a newer sample and repeated incidents appeared in both development and evaluation. The comparison is invalid, so the policy routes it to repair.
The rebuilt evaluation freezes versions, separates time periods, deduplicates incidents, and reports per-category errors, ambiguous cases, confidence behavior, correction time, fallback, and failure visibility. The language model retains higher average accuracy but invents unsupported rationale and misroutes more access incidents. The classifier is more stable in escalation.
Independent challenge finds overlapping service categories, making some labels ambiguous. The authorized disposition is limited comparison after taxonomy repair, not selection. Conditions require a frozen taxonomy, visible source date, escalation threshold, approved synthetic and historical data, human review, and a four-week decision date.
The evidence packet contains classification, alternatives, data/version manifest, repaired evaluation, error analysis, challenge, dissent, limited decision, release restrictions, monitoring, and review date. It does not establish that either method is safe, compliant, or generally superior.
Failure and hold conditions
Hold selection when context is unclear, the baseline is omitted, criteria differ by alternative, data lineage cannot reproduce, versions do not match, consequential errors are hidden by an average, challenge is not competent or independent, findings lack closure, an exception has no expiry, or release exceeds approved conditions.
Test a risk-based sample of selections, holds, rejections, changes, and expired decisions. Policy acknowledgement alone is not evidence that the policy operated or was effective.
Two Prune can place model-selection policy work inside a realistic AI-enabled assessment and evaluate how a participant defines requirements, compares evidence, challenges assumptions, manages exceptions, and communicates a defensible policy. It does not select production models, certify controls, or make employment decisions.
Sources
Board of Governors of the Federal Reserve System: SR 26-2 revised guidance on model risk management
- National Institute of Standards and Technology: AI Risk Management Framework Core
Sources: [1] [2]
Sources
- 1.Federal Reserve: SR 26-2 · www.federalreserve.gov
- 2.NIST: AI RMF Core · National Institute of Standards and Technology