Governing model risk in AI-enabled work

Define the model's role in the work

Start with the business process rather than the model label. Record inputs, transformations, outputs, users, downstream decisions, human review, and consequence. Distinguish a model that drafts language from one that ranks options, retrieves evidence, calculates a value, or triggers an action. The same model can create different risk in different uses.

List dependencies outside the model: data pipelines, retrieval indexes, prompts, tool permissions, external services, user instructions, and review capacity. Treat these as part of the operating system because failures can appear as plausible model output.

The Federal Reserve's 2026 revised guidance on model risk management emphasizes a risk-based approach tailored to a banking organization's model profile, size, complexity, and use. It supersedes SR 11-7. This is sector-specific supervisory guidance, not a universal rule for every organization, but its treatment of development, validation, effective challenge, governance, and model inventories is useful when responsibilities need separation.

Sources: [1]

Build the model-risk scenario ledger

For each scenario, record the use-case version, initiating condition, assumption under stress, expected safe behavior, observable evidence, test data, reviewer, severity, stop condition, remediation owner, retest result, residual uncertainty, and monitoring trigger. Include ordinary errors, degraded inputs, misleading context, misuse, and changes in downstream reliance.

Link every test to a decision. A test without an acceptance rule cannot support release. Define which failures block operation, restrict use, require warning or review, or can be tolerated temporarily. Retain failed versions and explain why a later result changes the disposition.

Assign challenge independently enough for the consequence. The person who designed a prompt can supply context, but should not be the only person deciding whether its failure modes are acceptable. Record disagreements, overrides, and evidence gaps so governance does not collapse into a sign-off checkbox.

The NIST Generative AI Profile identifies risks including confabulation, information integrity, privacy, intellectual property, harmful bias, and human-AI configuration. It is a voluntary companion to the AI RMF and should inform scenario selection without being treated as proof that a local control works.

Sources: [2]

Worked artifact: a synthetic demand-forecast narrative

Imagine a synthetic planning team using a language model to explain a statistical demand forecast. The model does not produce the forecast; it retrieves approved assumptions and drafts commentary. The scenario ledger includes conflicting source versions, missing regions, retrieval failure, unsupported causal language, and a sudden model-provider change.

In one test, the retrieval index contains both an approved pricing assumption and an older proposal. The draft confidently attributes higher demand to the older price. The output is numerically consistent with the forecast but evidentially wrong. The ledger records a release-blocking source-authority failure, not a generic hallucination.

The team adds version metadata, requires each material explanation to link to an approved assumption, and stops generation when two sources claim current authority. A retest activates the stop condition and routes the conflict to the planning owner. This demonstrates the mechanism in a synthetic case; it does not establish effectiveness under production volume.

Verify operation and change control

Reperform selected tests with the deployed model, prompt, retrieval set, permissions, and interface. Confirm that logs preserve the evidence needed to reconstruct decisions and that reviewers can stop release. Compare documented limits with actual user behavior and downstream reliance.

Reopen the ledger after changes to model version, data, prompt, tools, user population, output format, decision consequence, or monitoring result. Report not tested and unavailable evidence explicitly. Two Prune can provide task-bounded evidence about participant behavior in simulations; it does not validate customer models, attest to controls, or decide employment outcomes.