Practical guide

How to operationalize quality assurance in AI workflows

Operational quality assurance for AI-enabled work is a repeatable path from requirement to evidence to disposition. A policy can state expectations, but the workflow must show who selects cases, who tests, how defects are classified, what evidence is retained, and who can restrict or stop operation. A dashboard or average score alone is not an operating control. This educational workflow needs.

By Two Prune

Operational quality assurance for AI-enabled work is a repeatable path from requirement to evidence to disposition. A policy can state expectations, but the workflow must show who selects cases, who tests, how defects are classified, what evidence is retained, and who can restrict or stop operation. A dashboard or average score alone is not an operating control. This educational workflow needs adaptation to the organization’s sector, jurisdiction, contracts, data rules, and risk tolerance. It does not establish legal compliance or prove that a control is effective outside the evidence reviewed.

Define the control objective

Use a bounded objective such as: “Before consequential use and after material change, provide reproducible evidence that the AI-enabled workflow meets its stated requirements, exposes material defects, and routes an authorization decision to the accountable owner.”

Name the covered workflow, systems, model or service, configurations, source collections, users, data classes, geographies, and final actions. Record exclusions with a rationale and review date. Include human-edited AI output when it can still affect the final work.

The NIST AI Risk Management Framework says AI systems should be tested before deployment and regularly while operating. It calls for documented measurement, uncertainty, formal reporting, and independent review where useful. NIST presents the framework as voluntary and adaptable, not a certification. NIST AI RMF Core

Sources: [1]

Run the PROOF loop

PROOF turns the objective into an operating procedure with evidence at every handoff.

### P: Parse requirements into defect classes

The workflow owner creates a versioned requirement register. For each requirement, record the source, reason, affected outcome, consequence of failure, verification method, and accountable owner. Convert vague requirements into observable conditions.

For a sourced briefing workflow, separate source completeness, citation accuracy, distinction between fact and inference, date relevance, calculation correctness, confidentiality, required caveats, and escalation. Do not collapse these into “quality.” Define critical, major, and minor defect classes before testing. A critical unsupported claim must remain visible even if most cases pass.

### R: Risk-select and freeze the test

The assurance owner selects representative, boundary, changed-context, adversarial, accessibility, recovery, and known-failure cases in proportion to consequence. Record the case-selection rationale and the contexts not covered. Freeze the workflow version, configuration, model or service, approved sources, prompts or instructions, data transformations, expected result, evaluator guide, and acceptance rules.

Missing coverage is an evidence limitation, not a zero defect count. If the cases exclude a language, data type, user group, integration, or high-consequence path, show that exclusion in the authorization record.

### O: Observe execution and preserve evidence

The evaluator runs the frozen cases and records inputs, outputs, source traces, checks performed, defects, uncertainty, deviations, and environment details. Automate deterministic checks where useful, but preserve human review for requirements involving context, judgment, communication, or harm.

Use an evidence identifier that connects the requirement, case, result, defect, repair, and retest. Minimize personal or sensitive data and apply the organization’s access and retention rules. Record unavailable evidence rather than reconstructing it from memory.

The GAO AI Accountability Framework organizes oversight around governance, data, performance, and monitoring, with questions and audit procedures for entities and assessors. It is a primary-source accountability model, not proof that a particular workflow complies with law or performs adequately. GAO AI Accountability Framework

### O: Own the disposition and exceptions

The decision authority reviews the evidence packet and chooses one recorded disposition:

Every exception needs an owner, rationale, compensating control, affected scope, approval, and expiry. A failed material requirement cannot be relabeled as an accepted limitation by the delivery team alone. If the evidence owner and workflow owner disagree, route the documented disagreement to the decision authority.

### F: Follow operation and feed change back into testing

Monitor material defects, overrides, complaints, escalation failures, source outages, drift, latency, unauthorized use, and exception age as relevant. Give each signal a review cadence and action rule. A threshold should lead to investigate, restrict, repair, pause, or retire, not merely a colored chart.

Trigger targeted reassessment after a material change to model, prompt, source set, data, integration, user population, decision context, or policy. Also run periodic review when no incident is reported. Feed verified production defects into the frozen regression set while retaining the original evidence lineage.

  • authorize within the tested boundary
  • authorize with restrictions and an expiry
  • repair and retest before use
  • pause an operating workflow
  • reject or retire the use case

Sources: [2]

Required handoffs and artifacts

| Handoff | Accountable role | Evidence received | Evidence produced | Failure route | | --- | --- | --- | --- | --- | | scope to requirements | workflow owner | intended use and boundary | requirement and defect register | return for boundary repair | | requirements to test | assurance owner | register and risk map | frozen test plan and cases | hold missing material coverage | | test to review | evaluator | cases, guide, environment | execution results and defect log | preserve deviation and escalate | | review to decision | decision authority | evidence packet and uncertainty | authorization or restriction record | repair, pause, reject | | operation to reassessment | workflow owner | monitoring and change signals | review record and retest request | restrict or stop |

The minimum packet should contain a workflow and data-flow map, versioned requirement register, risk-control-test matrix, frozen test plan, case manifest, execution records, defect and exception registers, reviewer notes, disposition, change log, monitoring reviews, and next review date.

Test whether the control actually operates

Select one material requirement and perform a vertical trace:

Then perform a failure injection. Introduce a stale source, missing citation, incorrect calculation, prohibited data class, or unavailable reviewer. Confirm that the workflow creates the expected defect, prevents an unapproved action, and routes the issue to someone with authority. If the failure is only visible in a dashboard after the final action, the preventive or detective boundary may be too late.

  • find its authoritative source and current version
  • locate the mapped control and frozen case
  • reproduce the test result
  • follow any defect through repair and retest
  • confirm that the decision authority saw the result
  • verify that the authorization boundary matches the tested context
  • inspect the next review or change trigger

Worked example: research briefing workflow

A consulting team uses an AI assistant to assemble a client briefing from approved research. The assistant cannot publish, but the briefing contains external claims and recommendations.

Parse: requirements cover approved-source use, citation accuracy, date relevance, distinction between source fact and inference, calculation replay, confidentiality, and required caveats. Unsupported decision-relevant claims and confidentiality breaches are critical defects.

Risk-select: the assurance owner freezes twelve representative cases, including conflicting sources, an obsolete report, a calculation, a missing caveat, and an attempted unapproved source. The test does not cover other languages, so the authorization record excludes them.

Observe: an evaluator independent of the draft author runs the cases. Ten are acceptable. One contains an unsupported material claim and another omits the newer conflicting source. The overall completion rate is not used to hide those defects.

Own: the decision authority chooses authorize with restrictions. Every claim must be source-checked, a senior reviewer approves each briefing, and retrieval filters must be repaired and retested within 30 days. The exception has an owner and expiry.

Follow: production review tracks unsupported-claim defects, source-link failures, overrides, review time, and source-set changes. A new retrieval collection triggers targeted reassessment. The operating evidence supports a restricted decision for this workflow version; it does not prove universal quality or compliance.

Two Prune can use assurance work as a realistic assessment scenario and describe task-bounded evidence across framing, evidence navigation, AI use, verification, judgment, and communication. It does not certify control effectiveness, provide legal advice, promise safe operation, or make employment decisions.

Sources

NIST AI Risk Management Framework Core

  • U.S. Government Accountability Office: Artificial Intelligence Accountability Framework

Sources: [1] [2]

Sources

  1. 1.NIST AI RMF Core · National Institute of Standards and Technology
  2. 2.GAO AI Accountability Framework · U.S. Government Accountability Office