How to monitor model drift in AI-enabled work

Define the monitored decision

Start with the work outcome that matters: a classification, summary, recommendation, calculation, or drafted action. Record the intended users, input boundaries, acceptable output conditions, human review, and consequence of error. The NIST AI RMF Core calls for production monitoring and documented test, evaluation, verification, and validation methods.

For an AI-assisted policy summarizer, a monitored decision might be whether an operations manager can rely on the summary to identify changed obligations. The boundary includes the policy version, jurisdiction, effective date, source hierarchy, and requirement to cite decisive text.

Do not label every variation drift. Some variation is expected; the question is whether it changes the task result or the control needed.

Sources: [1]

Build a reference packet and trigger register

Create a small frozen packet of representative, difficult, and previously failed cases. Preserve the source files, expected decision, reviewer rationale, configuration, prompt or workflow version, and known limitations. Add a trigger register for model updates, connector changes, source-schema changes, new user populations, policy revisions, and material incident patterns.

The NIST AI RMF Measure playbook recommends documenting test sets, metrics, tools, and measurement processes. A reference packet supplies that traceability at the workflow level.

Run both scheduled and triggered checks. A quarterly calendar alone may miss a major provider update; update notices alone may miss gradual source or user change.

Sources: [2]

Work a synthetic drift ledger

Suppose the policy summarizer has six frozen cases. After a model update, five still meet the expected conditions. In the sixth, the output now omits an exception clause and moves the citation to a general introduction.

The drift ledger records:

Keep qualitative differences beside numerical measures. A stable average score can hide one consequential reversal.

• D-01: reference case P-06, exception handling
• Baseline: exception quoted, effective date stated, decision held for specialist review
• Current: exception omitted, confident recommendation issued
• Configuration change: model version changed; prompt and sources unchanged
• Decision impact: material because the recommendation reverses
• Disposition: pause this workflow configuration, add an exception-preservation test, and route to the policy owner

Separate four kinds of change

Model change covers altered behavior from the model or its configuration. Data change covers new distributions, formats, missingness, or source quality. Workflow change covers prompts, tools, retrieval, user instructions, and human review. Context change covers the decision, population, policy, or consequence of error.

The GAO AI Accountability Framework groups practices under governance, data, performance, and monitoring. Use those categories to prevent a detected difference from being attributed to the model before other causes are checked.

For each meaningful change, reproduce the result, isolate one variable where possible, inspect decisive source links, and have an accountable reviewer judge task impact. Record “cause unknown” when evidence cannot distinguish explanations.

Sources: [3]

Connect monitoring to action

Define responses before alerts fire: observe, narrow use, add review, revert configuration, suspend the task, or redesign the reference packet. Name who can take each action and what evidence supports restoration.

Review the packet when the task itself changes. A frozen benchmark that no longer represents current work can provide stable numbers while monitoring the wrong thing.

Two Prune can create a consistent assessment task and retain the submitted artifact plus available workflow evidence for human interpretation. It does not monitor production models or establish a universal drift threshold. When an assessment changes, score interpretation should consider scenario, tool, and evidence differences rather than assuming participant performance caused the change.