Continuous Improvement for AI-Enabled Work: Maintain a Failure-to-Change Ledger

Capture the failure as observed

Record the date, task, artifact version, evidence, consequence, and detection route. Describe what happened without assigning a cause prematurely. Distinguish user behavior, source defects, model behavior, interface design, policy ambiguity, and missing review. One event may involve several categories, but each proposed cause should have a test.

The CDC’s 2024 evaluation framework describes a practical cycle that assesses context, focuses questions, gathers credible evidence, generates conclusions, and learns from and uses insights (CDC approach to programme evaluation). It supports connecting evidence to action without treating every operational review as formal research.

Sources: [1]

Write one change hypothesis

State the smallest change expected to reduce the failure, why it should work, where it applies, and what unintended effect to watch. Assign an owner, test population, start date, evidence window, and rollback condition. Avoid combining training, model, prompt, data, and policy changes unless the decision genuinely requires a package and attribution is not needed.

The 2025 GAO Green Book identifies monitoring as part of an internal-control system and describes evaluating whether controls are designed, implemented, and operating (GAO Green Book). That supports checking operation over time. This ledger is an educational adaptation and does not establish conformity with the Green Book.

Sources: [2]

Worked artifact: citation failure cycle

Imagine a synthetic research workflow where summaries repeatedly cite secondary articles instead of the required primary source. The ledger records three examples and rejects the early explanation that users simply need better prompts. Inspection shows the interface hides source type and the checklist asks only whether a citation exists.

The team changes the checklist to require source classification and adds a visible primary-source field. A controlled set of twelve tasks tests the change. Reviewers measure whether claims trace to appropriate sources and whether completion time or abandonment worsens. The result shows improvement on the test set but two ambiguous government mirrors remain, so the change is retained with a follow-up classification rule.

Verify beyond the first success

Repeat the check on a different task, reviewer, and time period. Monitor whether people bypass the new step or enter meaningless values. Compare like populations and preserve logging changes. A successful test supports the bounded change; it does not prove a general productivity or quality outcome.

Close, extend, or reverse

At the review date choose close, extend, modify, or roll back. Record the evidence and accountable decision. Promote confirmed practices into the workflow documentation and retire obsolete instructions. Keep unresolved causes visible so they are not rediscovered as new incidents.

Control the improvement portfolio

Maintain a view of active changes so several teams do not alter the same workflow at once. Record dependencies, overlapping populations, shared measures, and freeze periods. If changes cannot be isolated, state that attribution is limited and focus on whether the combined package is acceptable for the decision.

Set capacity for reviewing old changes as well as launching new ones. An experiment that never reaches a close, extend, or reverse decision creates operational debt. Archive abandoned work with its evidence and reason, so a later team can see why it stopped without treating the old hypothesis as established practice.

Review the ledger for selection bias. Easy-to-detect failures may receive repeated attention while rare, consequential, or poorly logged failures remain invisible. Add periodic risk-based sampling and a route for frontline staff to report near misses. Record the limits of the intake process alongside improvement counts.

This method is educational. It does not establish causal impact, control effectiveness, or a mature benchmark. Organizations need current operating evidence, policies, and qualified owners for consequential changes.