Practical guide

How to operationalize model selection in AI workflows

Operational model selection is a controlled sequence that connects a real use to comparable alternatives, reproducible evidence, independent challenge, an authorized decision, an exact release, and continued-fit monitoring. The workflow should stop teams from selecting a method because it is already licensed, wins one benchmark, produces an impressive demonstration, or is described as a.

By Two Prune

Operational model selection is a controlled sequence that connects a real use to comparable alternatives, reproducible evidence, independent challenge, an authorized decision, an exact release, and continued-fit monitoring. The workflow should stop teams from selecting a method because it is already licensed, wins one benchmark, produces an impressive demonstration, or is described as a temporary experiment.

Source and applicability boundary

Federal Reserve SR 26-2 is current supervisory guidance for specified banking organizations. It emphasizes purpose, intended use, risk-based rigor, development and testing, effective challenge, validation, limitations, monitoring, and governance; it expressly excludes generative and agentic AI from its model definition. The NIST AI Risk Management Framework Core is voluntary and provides broader governance outcomes for mapping, measuring, and managing AI risks. The workflow below adapts useful control concepts without claiming those sources legally apply to every organization or system. Federal Reserve: SR 26-2 NIST: AI RMF Core

Sources: [1] [2]

Run selection through ROUTE

### R — Request, purpose, and risk route

The requester submits the use, decision consequence, affected users, data, output, human role, fallback, required integrations, prohibited uses, materiality, and named business owner. The governance owner classifies the method and assigns the applicable route.

Classification choices can include statistical model, machine-learning model, generative AI, agentic system, deterministic rule, spreadsheet, vendor service, or other decision-support method. An excluded category must enter another named governance path; exclusion is not exemption.

The request is held when purpose is vague, the owner cannot authorize the use, the fallback is absent, or a “pilot” will affect real decisions without controlled limits.

### O — Options, criteria, and evidence plan

List the current process or no-change baseline and materially credible alternatives. These may include process redesign, clearer information, deterministic rules, human-supported review, a simpler model, a different vendor, or no automation.

Set common criteria before final evaluation. Include task quality, consequential error types, data suitability, uncertainty, failure visibility, human burden, accessibility, privacy, security, operational fit, cost, latency, maintainability, portability, and exit. Mandatory requirements are gates, not weighted preferences.

Create an evidence plan naming data authority, sample and time boundary, versions, configuration, environment, measures, thresholds, scenarios, reviewers, uncertainty treatment, and acceptance records. The plan must make a fair comparison possible; unequal data or different definitions invalidate a ranking.

### U — Undertake evaluation and independent challenge

Freeze candidate versions and run the evaluation. Retain inputs, exclusions, transformations, prompts or system instructions where relevant, configurations, outputs, reviewer decisions, calculations, and issue logs. Report average performance alongside consequential errors, relevant subgroups or contexts, boundary cases, missing inputs, and fallback behavior.

An independent challenger checks conceptual fit, alternative coverage, data relevance, leakage, labels, measures, threshold choice, reproducibility, implementation, limitations, and contradictions. Independence requires competence, access to evidence, authority to disagree, and freedom from being the sole author or commercial advocate.

Every finding records severity, consequence, owner, disposition, due date, and closure test. Preserve dissent. “Reviewed” is not a disposition.

### T — Tie approval to the exact release

The authorized approver chooses approve, limited test, repair, hold, reject, or retire. The record names the exact version, configuration, users, approved and prohibited uses, human review, fallback, conditions, residual uncertainty, monitoring, and expiry.

Before release, an operator verifies that the deployed version, instructions, data connections, access, thresholds, fallback, logging, and user communication match the decision. A release mismatch reopens the gate.

Material changes to vendor, model, data, prompt architecture, tools, integration, population, use, consequence, jurisdiction, or control enter triage before deployment. The owner decides whether evidence must be updated, challenged again, approved again, restricted, or stopped.

### E — Exceptions, monitoring, and exit

An exception states exact scope, reason, authority, compensating control, evidence, expiry, and stop trigger. Repeated renewal cannot silently create permanent approval.

Monitor signals tied to the approved use: input changes, error patterns, overrides, incidents, complaints, performance drift, unavailable controls, fallback use, user workarounds, and material upstream changes. Monitoring thresholds route to investigate, restrict, repair, revalidate, switch, or retire.

Retain the request, classification, alternatives, evidence plan, version manifest, evaluation, calculations, challenge, findings, decision, release match, exceptions, monitoring, incidents, changes, and exit evidence.

Operating handoffs and accountable records

| Workflow gate | Accountable role | Required record | Hold condition | | --- | --- | --- | --- | | Request | business owner | use and consequence brief | no authorized owner | | Classification | governance owner | scope and risk route | unresolved category | | Evidence plan | evaluation owner | alternatives and common test | incomparable candidates | | Evaluation | technical and data owners | reproducible results packet | version or data mismatch | | Challenge | independent reviewer | findings and dispositions | material issue open | | Decision | authorized approver | exact decision and conditions | authority or limits missing | | Release | release owner | approved-to-deployed match | configuration differs | | Continued fit | operational owner | monitoring and change log | threshold breached |

Use a workflow state machine: requested → classified → planned → evaluated → challenged → decided → release-verified → monitored → retired. Repair returns to the affected prior gate. Hold stops progression but preserves evidence.

Worked workflow: service-request routing

A synthetic operations team proposes a general-purpose language model to route internal service requests. A demonstration reports 91 percent accuracy and claims it beats the current supervised classifier.

Request: the business owner defines the consequence as routing access, account-provisioning, device, and payroll questions to the correct queue. Payroll and access errors have higher consequences. Human triage remains the fallback.

Options: ROUTE adds the current human-assisted process, improved keyword rules, the existing classifier, and the language model. Common criteria include category error, high-consequence error, ambiguous-case escalation, unsupported rationale, correction time, latency, cost, failure visibility, and fallback operation.

Evaluation: the manifest reveals that the language model used a newer sample and repeated incidents appear across development and evaluation. The 91 percent comparison is held. The team freezes time periods, deduplicates incidents, repairs category definitions, and tests all alternatives on the same set.

The repaired results show the language model has higher average routing accuracy but invents explanations and misroutes more access incidents. The classifier is less accurate overall but more stable on escalation. Keyword rules are transparent and handle a narrow high-volume subset. Human-assisted routing is slower but remains the safest fallback for ambiguous requests.

Challenge: an independent reviewer finds that “account access” and “account provisioning” overlap in the operating taxonomy, so several labels cannot support a definitive winner. The reviewer also finds that the proposed production prompt omits an escalation instruction used in evaluation.

Decision: the approver selects repair taxonomy and run a limited shadow comparison, not full release. Conditions require an approved taxonomy, version-locked candidates, production-equivalent instructions, visible source dates, an escalation threshold, human confirmation, no automated downstream action, and a four-week review.

Release match and monitoring: the operator compares hashes and configuration, confirms shadow mode, tests fallback, and records access. Monitoring tracks high-consequence errors, unsupported explanations, overrides, ambiguous cases, latency, and missing logs. A missing escalation instruction or access-error threshold breach stops the comparison.

The artifact contains the request, classification, alternatives register, common criteria, data and version manifest, repaired evaluation, error analysis, challenge and dissent, limited decision, release-match record, monitoring thresholds, exception register, and review date. It does not establish that any method is safe, compliant, or generally superior.

Failure and escalation conditions

Stop the workflow when alternatives are token, selection criteria change after results, evaluation and release versions differ, leakage or unequal samples invalidate comparison, consequential errors are averaged away, the challenger lacks authority or evidence, a condition has no owner, or monitoring cannot detect an approval breach.

AI can help assemble manifests, compare records, or flag candidate discrepancies. Human owners remain responsible for applicability, criteria, evidence quality, challenge, approval, and operating action.

Two Prune provides realistic AI-enabled work assessments and evidence about observed participant performance. It may help an organization test whether people can frame tasks, navigate evidence, use AI, verify outputs, exercise judgment, and communicate. It does not select production models, certify controls, promise safe operation, or make employment decisions.

Sources

Federal Reserve: SR 26-2

  • NIST: AI RMF Core

Sources: [1] [2]

Sources

  1. 1.Federal Reserve: SR 26-2 · www.federalreserve.gov
  2. 2.NIST: AI RMF Core · National Institute of Standards and Technology