Practical guide
AI capability pilot design for an operations excellence leader
An operations excellence pilot should test one uncertain mechanism in a bounded process, not stage a broad demonstration. The operations excellence leader may own the improvement method and evidence packet, while the process owner owns the operating outcome, technology owns the configured system, risk specialists review their domains, and an authorized sponsor makes the expand, repair, narrow,.
An operations excellence pilot should test one uncertain mechanism in a bounded process, not stage a broad demonstration. The operations excellence leader may own the improvement method and evidence packet, while the process owner owns the operating outcome, technology owns the configured system, risk specialists review their domains, and an authorized sponsor makes the expand, repair, narrow, hold, or stop decision. The useful output is a pre-registered pilot protocol that compares credible alternatives, exposes realistic failure conditions, measures displaced work, and preserves enough evidence to explain the result.
Use the TRIAL pilot method
TRIAL turns a promising business case into an executable, reversible test.
### T — Target one constraint and riskiest assumption
Choose one end-to-end process, user or customer outcome, and unit of analysis. Reconstruct current work from system data, sampled cases, frontline observation, user evidence, quality review, and exceptions. State the mechanism the proposed AI contribution is expected to change.
Write the riskiest assumption as a falsifiable statement. For example: “When the complete case file is available, participants using AI assistance can produce a source-linked exception rationale without increasing missed policy conflicts or reviewer effort.” Avoid testing adoption, satisfaction, quality, time, and business impact in one small exercise.
The 2026 Magenta Book Test and Learn annex recommends starting with the assumptions that are most critical and least supported, choosing methods proportionate to risk and potential harm, and using early tests to build evidence before scaling. It is public-sector guidance; here it supplies a disciplined testing pattern rather than proof about a private workflow. GOV.UK: Test and Learn
### R — Roles, comparison, and risk boundary
Classify the operations excellence leader as owner, reviewer, adviser, or collaborator for each pilot decision. Name the process owner, participant, frontline representative, technology and data owners, reviewer, privacy and security contacts, finance partner, and person authorized to pause the test.
Compare at least three credible routes: the current process, process improvement without generative AI, and the AI-assisted design. Add deterministic automation when it could address the same constraint. Record allowed tools and data, prohibited uses, human decisions, escalation, stop conditions, and what happens to pilot records afterward.
NIST's AI Risk Management Framework Core calls for documented context, intended use, human oversight, experimental design, representative evaluation conditions, test sets, metrics, independent input, uncertainty, monitoring, and generalizability limits. Use these outcomes to review the protocol; framework alignment does not show that a local pilot is effective. NIST AI RMF Core
### I — Instrument realistic capability tasks
Build cases that require the actual capability in question: framing ambiguity, finding controlling sources, using AI in an inspectable way, checking a calculation or claim, detecting conflicting evidence, escalating uncertainty, revising, and delivering a usable artifact. Include ordinary, edge, and adverse cases.
Freeze the source pack and expected evidence anchors. Add a designed challenge that a superficial workflow is likely to miss, such as a stale policy, changed threshold, duplicated case, unit mismatch, conflicting attachment, or authority gap. Preserve the final artifact, material AI contribution, verification action, reviewer result, and timing needed to diagnose the mechanism.
Do not collect irrelevant keystrokes or personal content. Instrumentation should be proportionate and governed. A complete evidence packet is one that answers the pilot decision, not one that records everything a person did.
### A — Allocate cases and calibrate review
Define the eligible case population, sample method, inclusion and exclusion rules, comparison assignment, participant experience, tool configuration, and analysis plan before results are known. If random allocation is infeasible, explain the comparison and likely confounds. Do not compare easy AI cases with difficult baseline cases.
Use an anchored rubric. Calibrate at least two reviewers on paired examples containing a clear pass, clear failure, and ambiguous case. Record disagreements, adjudication, and rubric changes. Keep the original results when the rubric changes; do not silently rescore only weak cases.
Pre-register measures across distinct layers:
Access, usage, self-reported confidence, observed capability, control operation, process quality, and business outcome are different measures.
### L — Learn at the decision gate
Write thresholds and prohibited conditions before the test. Report distributions and cases, not only averages. Compare alternatives, inspect failures, and state whether the evidence supports expand, repair and retest, narrow, hold, or stop.
GAO's AI accountability work identifies planning, monitoring, traceability, corrective action, assessment of utility in the current context, and explicit conditions for expanded use. That supports a decision gate and post-pilot monitoring plan without implying that a pilot establishes durable operating performance. GAO: Key practices for accountable AI
An expand decision should still define a bounded next stage, monitoring, change triggers, an exit route, and evidence that remains unproven. Preserve null and adverse results.
- artifact quality and source traceability;
- contradiction and exception detection;
- verification action and escalation correctness;
- review effort, rework, and displaced work;
- control failures and incidents;
- missing records and protocol deviations.
Sources: [1] [2] [3]
Copyable pilot protocol
| Protocol field | Required entry | | --- | --- | | Decision | Exact decision, owner, date, and available dispositions | | Constraint | End-to-end process, outcome, unit, baseline, exception pattern | | Riskiest assumption | Falsifiable mechanism statement and failure signal | | Alternatives | Current, process repair, deterministic option, AI-assisted option | | Roles | Owner, reviewer, adviser, collaborator, pause authority | | Cases | Population, sample, ordinary/edge/adverse mix, frozen sources | | Challenge | Planted conflict, stale source, calculation trap, or authority gap | | Measures | Quality, traceability, verification, escalation, effort, missingness | | Review | Rubric anchors, calibration cases, independent rating, adjudication | | Controls | Allowed tools/data, human decision, incidents, stop conditions | | Gate | Expand, repair, narrow, hold, stop thresholds | | Exit | Record treatment, alternative process, owner, and change triggers |
Common objections and reporting boundaries
“The tool is already popular” does not answer the pilot decision. “The average score improved” can hide unsafe failures. “A simulation is not live work” is correct: use it for controlled capability evidence and keep operational conclusions bounded. “The operations excellence leader owns transformation” does not grant sole authority over process outcomes, data, technology, risk, or budget.
Report the workflow, case population, alternatives, configuration, observation window, protocol deviations, missingness, incidents, reviewer disagreement, review effort, and transfer limits. Reopen the protocol when the source environment, process, model, prompt, interface, population, or consequence changes.
Two Prune boundary
Two Prune can run realistic work simulations and provide evidence about observed performance in context framing, evidence navigation, AI orchestration, verification and risk control, judgment and synthesis, and communication. It does not operate a customer's live pilot, establish process impact or ROI, assign budgets, certify controls, or make employment decisions. Customers retain responsibility for process design, systems, data, operational testing, approvals, and use of assessment evidence.
Sources
- 1.GOV.UK: Test and Learn · UK Government
- 2.NIST AI RMF Core · National Institute of Standards and Technology
- 3.GAO: Key practices for accountable AI · U.S. Government Accountability Office