Quality assurance for AI-enabled knowledge work

Bound the release decision

Select one workflow and output, such as an assistant-generated policy summary for internal staff. Define the intended users, permitted decisions, authoritative sources, freshness requirement, data boundary, review owner, and consequence of a wrong answer. State what the release does not cover.

Translate quality goals into observable requirements. Examples include traceable claims, current-source selection, correct calculations, mandatory-topic coverage, appropriate uncertainty, safe failure, accessibility, and escalation when authority is missing. Define release-blocking defects separately from preferences so fluent wording cannot compensate for a material error.

The NIST AI Risk Management Framework Core describes mapping context, measuring risks and impacts, documenting testing, seeking independent review, responding to findings, and monitoring after deployment. It supports a structured assurance process but does not certify a particular workflow.

Sources: [1]

Assemble the release assurance case

For each material requirement, record the risk, control, owner, test population, expected result, observed result, evidence link, defect severity, correction, retest, residual uncertainty, approver, and expiry trigger. Include ordinary cases, boundary cases, missing inputs, conflicting evidence, malformed content, permission errors, and downstream reuse.

Keep development evidence separate from operational evidence. A pre-release benchmark may show controlled performance, while production monitoring reveals changed users, data, integrations, or incentives. Record where the evidence came from and which decision it can support. A pass in one environment should not silently authorize a different use.

Assign independence proportionately. The person who built a control may demonstrate it, but a separate reviewer should challenge decision-critical evidence where consequence warrants it. Record conflicts of interest, unavailable reviewers, and reduced assurance rather than describing every check as independent.

NIST's 2026 report on challenges to monitoring deployed AI systems describes post-deployment monitoring as a broad and fragmented field and highlights variability in deployed AI behavior. This supports maintaining operational evidence rather than treating launch as the final assurance event.

Sources: [2]

Worked artifact: a synthetic policy-summary release

Imagine a synthetic internal assistant that summarizes travel policy. The assurance case requires every material entitlement and restriction to cite the current approved document, label ambiguous cases, and route exceptions to the policy owner. Tests include ordinary travel, an expired exception, two conflicting versions, missing receipts, and a user without access to the underlying document.

The first run answers all ordinary cases fluently but uses the obsolete policy when filenames are similar. Because source currency is release-blocking, the team stops the release, changes document precedence, and adds version metadata to the output. A retest catches another issue: users without document access can still see quoted detail in the generated response.

The access control is corrected and the full varied set is rerun. The release owner approves only the internal, authenticated workflow and records that international tax questions remain outside scope. The example is synthetic and does not demonstrate universal control effectiveness, legal compliance, or Two Prune customer outcomes.

Maintain assurance after release

Monitor failures, overrides, escalations, source changes, permission drift, reviewer workload, and user-reported harm. Reopen the assurance case after a material model, data, integration, policy, audience, or consequence change. Retain failed tests and decision rationales so regressions are visible.

Set an expiry for the release decision. An assurance case should identify what evidence is monitored continuously, what is sampled periodically, and which signal pauses use immediately. A quiet incident log does not by itself establish that the system is working as intended.

Two Prune can create controlled opportunities to observe verification and risk-control behavior in a simulation. It does not operate customer quality controls, certify a system, or replace accountable technical, legal, privacy, security, and functional review.