Practical guide
What evidence shows model-risk controls are operating?
Evidence of an operating model-risk control must connect a defined model use to its inventory record, approval conditions, independent challenge, monitoring results, exceptions, and accountable action. A policy, validation report, or dashboard alone is design evidence. The decisive question is whether current records show that people detected, escalated, and resolved risk under representative.
Evidence of an operating model-risk control must connect a defined model use to its inventory record, approval conditions, independent challenge, monitoring results, exceptions, and accountable action. A policy, validation report, or dashboard alone is design evidence. The decisive question is whether current records show that people detected, escalated, and resolved risk under representative and adverse conditions.
Set the evidence boundary
Define the control objective in operational terms: models are used only for documented purposes, within approved limits, with current challenge and monitoring, and material deviations trigger a named response.
Record the scope before sampling:
The 2026 U.S. interagency model-risk guidance describes risk-based practices for model development, use, validation, monitoring, governance, inventory, documentation, and third-party products. Its stated scope is banking organizations and it explicitly excludes generative and agentic AI models, although organizations may use their own governance for tools outside scope. NIST’s AI Risk Management Framework offers broader voluntary outcomes for documentation, roles, oversight, monitoring, incident response, and third-party risk. Use these as authoritative reference points, not as a universal legal test or proof that a control is effective. Federal Reserve: SR 26-2 revised guidance on model risk management NIST AI RMF Core
- model or system identifier, version, purpose, owner, and users;
- decision supported and potential consequence of error;
- model class, materiality, and applicable internal policy;
- data, assumptions, dependencies, vendor components, and approved use limits;
- validation status, monitoring period, and open findings;
- accountable control owner, independent reviewer, and decision authority.
Sources: [1] [2]
Test the operating CHAIN
CHAIN follows evidence from the control population to the action taken. The links should agree; a polished record in one link cannot repair a break elsewhere.
### C — Coverage and classification
Reconcile the population of models and material model-like systems against inventories, deployment records, procurement records, workflows, and change logs. Sample both registered items and sources that could reveal unregistered use.
For each sample, test whether purpose, owner, materiality, users, dependencies, and approved boundaries match current operation. Record omissions and misclassification as findings rather than silently correcting the sample frame.
### H — History and authorization
Trace the current version through development or acquisition, challenge, approval, release, and material change. Confirm that approvals identify the exact version and conditions, and that exceptions have scope, approver, expiry, and review trigger.
Look for evidence of actual decisions: rejected releases, narrowed uses, temporary limits, remediation, or retirement. A system in continuous use with only positive approvals provides weak evidence that the escalation path works.
### A — Adverse challenge and validation
Inspect whether challenge matches purpose, methodology, data, materiality, and known failure modes. Evidence may include conceptual review, benchmark comparison, sensitivity analysis, outcome analysis, back-testing, alternative assumptions, data-quality tests, boundary cases, and independent reproduction.
Check who designed and performed the challenge, what evidence they could access, how disagreements were resolved, and whether material findings changed the model or its use. Independence is an operating property that must be visible in authority and behaviour, not only an organization chart.
### I — In-use monitoring and issue response
Select monitoring periods, including one with a threshold breach, data change, unusual outcome, user concern, or vendor change. Trace the alert to triage, impact assessment, containment, decision, remediation, communication, and closure.
Recalculate at least one monitored value from source evidence. Confirm that thresholds were defined before the event, or clearly identify when judgment was used. A dashboard screenshot without the underlying definition, population, or response is insufficient.
### N — Named decision and next review
For every exception or material finding, identify who decided to continue, limit, adjust, redevelop, replace, or retire the model. Record the rationale, evidence considered, residual risk, affected users, due date, and next test.
Verify closure rather than status labels. A finding marked closed should have remediation evidence and, where necessary, a retest showing that the observed failure was addressed.
Copyable evidence record
| Sample ID | Model / version / use | Expected control | Evidence inspected | Challenge or event | Observed action | Owner / approver | Disposition | Residual risk | Next test | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | MR-01 | exact identifier and decision | policy or procedure step | inventory, approval, validation, monitoring, issue record | adverse condition or operating period | traceable response | named people | pass / repair / limit / hold / retire | what remains unknown | date and method |
Attach the reconciled population, sample rationale, versioned evidence, recalculations, reviewer notes, disagreement log, exception records, and closure tests. Protect restricted model, data, security, and vendor information under the organization’s approved controls.
Worked test: demand forecast after a market shift
A synthetic planning team uses a weekly demand forecast to set inventory. The inventory record identifies version 4.2, approved for established products with eight weeks of history. Monitoring tracks absolute percentage error and bias by product family.
The coverage test finds that a new product launch also uses the forecast, even though it has only two weeks of history. The deployment log shows the route, but the inventory and approval do not. CHAIN records an unapproved use rather than expanding the approval after the fact.
The history test finds a temporary exception permitting manual overlays for four weeks. The exception names an approver but has no quantitative limit and expired six weeks ago. Planners still apply overlays. This is a break in authorization and exception closure.
The adverse-challenge sample selects the first week after a competitor exit. Forecast error crosses the monitoring threshold, but the dashboard aggregates established and launch products, masking the launch-family breach. A reviewer recalculates performance by product family and reproduces the issue.
The issue record shows that a planner raised the unexpected outputs in chat, yet no formal triage was opened. The control therefore fails in-use response even though the monthly dashboard was produced on time. The accountable owner limits version 4.2 to the approved established-product population, starts a launch-specific review, caps overlays under a documented rule, notifies affected planners, and schedules a retest after four weeks of data.
The final disposition is repair and limit, not “effective.” The evidence packet contains the unregistered-use finding, expired exception, disaggregated recalculation, user report, interim restriction, named decision, and retest plan. It does not claim that four weeks of later performance will establish suitability; that remains the next evidence gate.
Design evidence versus operating evidence
Classify artifacts accurately:
No single artifact establishes the whole control. Combine population coverage, representative samples, adverse cases, and longitudinal evidence. Increase the sample when you find unregistered use, stale approval, unreproducible monitoring, overdue findings, or silent scope expansion.
- a policy or procedure shows expected design;
- an inventory shows recorded scope at a point in time;
- a validation report shows the challenge performed on its stated version and use;
- a monitoring report shows measured conditions for a period;
- an alert-to-action trace shows response under an observed event;
- an expired or rejected exception can show that authority boundaries operated;
- a retest shows whether a specific remediation addressed the observed failure.
Hold and escalation conditions
Hold an operating conclusion when the population cannot be reconciled, the current version or use is unclear, validation evidence does not match production, monitoring cannot be recalculated, an exception lacks authority, a material breach has no response record, vendor limitations prevent necessary challenge, or remediation is marked closed without a supported retest.
Scope every conclusion to the models, uses, periods, controls, and samples examined. A passing sample does not certify an enterprise programme. Legal, regulatory, accounting, safety, and sector-specific conclusions require appropriately qualified review under the current rules that apply to the organization.
Two Prune can use a model-risk evidence task in a realistic AI-enabled work assessment and evaluate how a participant defines scope, navigates technical and governance records, challenges claims, handles exceptions, and communicates a defensible disposition. It does not validate production models, certify controls, or make employment decisions.
Sources
Board of Governors of the Federal Reserve System: SR 26-2 revised guidance on model risk management
- National Institute of Standards and Technology: AI Risk Management Framework Core
Sources: [1] [2]
Sources
- 1.Federal Reserve: SR 26-2 revised guidance on model risk management · www.federalreserve.gov
- 2.NIST AI RMF Core · National Institute of Standards and Technology