Practical guide

AI capability measurement guide for an operations excellence leader

An operations excellence leader needs an AI capability agenda that starts with process decisions, not tool enthusiasm. The role may own improvement methods and cross-functional coordination, but process owners, frontline teams, technology, data, risk, and finance retain distinct authority. The useful output is a decision agenda that shows which workflow questions require capability evidence,.

By Two Prune

An operations excellence leader needs an AI capability agenda that starts with process decisions, not tool enthusiasm. The role may own improvement methods and cross-functional coordination, but process owners, frontline teams, technology, data, risk, and finance retain distinct authority. The useful output is a decision agenda that shows which workflow questions require capability evidence, what alternatives exist, and what would justify expanding, repairing, narrowing, or stopping a change. Use the LOOP agenda: Locate the operational constraint, Own the decision roles, Open the alternatives, and Prove the next gate.

L — Locate the operational constraint

Choose a bounded end-to-end process and record its user or customer outcome, trigger, steps, handoffs, queues, sources, exceptions, rework, controls, and final decision. Separate elapsed time from touch time, waiting, review, and correction. An AI-assisted step may become faster while the full process becomes slower or less reliable.

Define the unit of analysis before examining results: case, completed journey, exception, handoff, or service outcome. Record volume, mix, missing records, reopened cases, and work that leaves the visible system. Use multiple sources: workflow data, sampled case reconstruction, user or frontline research, support contacts, quality review, and financial or resource evidence where appropriate.

The GOV.UK Service Manual says the measurement approach should match whether the team is assessing a transaction, an end-to-end journey, or contribution to a wider strategy, and recommends combining performance metrics with user research and other data rather than relying only on digital analytics. This is public-service guidance, not proof about a private workflow, but it supports an end-to-end evidence boundary. Measuring the success of your service

Sources: [1]

O — Own the decision roles

For each agenda item, classify the operations excellence leader as owner, reviewer, adviser, or collaborator. Then name the process owner, outcome owner, frontline participants, AI or technology owner, data owner, security and privacy reviewers, finance partner, and person authorised to pause the test. Do not infer budget or deployment authority from the role title.

The operations excellence leader may own process mapping, improvement cadence, measurement discipline, and comparison across workflows. The functional process owner should own the operating outcome and acceptable failure boundary. Technology owners govern the configured tool; risk functions review their domains; frontline teams contribute evidence about actual work and workarounds.

NIST's AI RMF Core calls for documented context, intended purpose, human oversight, roles and responsibilities, diverse input to risk decisions, testing, and ongoing measurement. Use those outcomes to make decision rights visible. Framework alignment does not demonstrate that a local AI workflow is effective. NIST AI RMF Core

Sources: [2]

O — Open the alternatives before selecting a pilot

Compare at least four routes:

Include staffing, policy, knowledge-management, and system-integration alternatives when they address the constraint. A capability assessment should not be used to justify an AI option whose process problem could be resolved more directly.

NIST's Baldrige operations guidance recommends designing work as end-to-end processes, integrating customer and supplier input, measuring requirements, finding defect sources, and regularly evaluating and improving processes. It is a general performance-improvement resource; it does not prescribe a particular AI intervention. NIST Baldrige: Operations

For each alternative, record expected benefit, new work, failure modes, dependencies, reversibility, evidence available, people affected, and decision owner. Keep the “do minimum” option credible so the AI-assisted route is not compared only with an artificially weak baseline.

  • keep the current process and make no change;
  • improve the process without generative AI, such as clearer rules, source cleanup, routing, or training;
  • use narrower automation or analytics for a defined step;
  • test an AI-assisted workflow with explicit review and stop conditions.

Sources: [3]

P — Prove the next gate with bounded evidence

Choose the smallest test that can change the decision. Capability evidence should require realistic work: framing the case, finding authoritative sources, using AI inspectably, detecting planted contradictions, checking outputs, escalating uncertainty, revising, and delivering a usable artifact. Preserve the final output and the evidence needed to understand the process.

Pre-register the gate:

Report distributions, exceptions, reviewer disagreement, missing records, review effort, and displaced work. Separate access and usage from observed capability, process quality, and business outcomes. The US Government Accountability Office's AI accountability framework organizes evidence around governance, data, performance, and monitoring and provides review questions for managers and assessors. Use it to structure an evidence packet, not to claim external approval. GAO AI Accountability Framework

  • expand when the named evidence is sufficiently complete and the workflow meets its bounded quality, control, and resource conditions;
  • repair and retest when the concept remains plausible but materials, configuration, measurement, supervision, or capability break down;
  • narrow when evidence supports only a smaller task, population, source type, or risk class;
  • hold when missing evidence prevents a sound decision;
  • stop when a prohibited condition occurs or the alternative no longer addresses the operational constraint.

Sources: [4]

Role-specific decision agenda

| Agenda decision | Leader role | Evidence needed | Alternatives to compare | Gate | | --- | --- | --- | --- | --- | | Which process constraint merits investigation? | owner or adviser | end-to-end journey, defects, queues, rework, user evidence, consequence | no change, source cleanup, policy or training, system repair | constraint and decision unit are explicit | | Is AI relevant to the constraint? | reviewer or collaborator | task decomposition, authoritative sources, variability, judgment points, data boundary | rules automation, analytics, workflow redesign, staffing | mechanism addresses the observed constraint | | What capability must people demonstrate? | collaborator with process owner | representative tasks, source use, verification, exception handling, communication | current supervision, knowledge check, guided practice | task evidence matches real failure modes | | Should a controlled pilot run? | owner of improvement method; process owner approves operations | baseline, comparison, protocol, owners, controls, stop conditions | do minimum, narrow automation, process-only pilot | protocol is decision-ready and reversible | | Should the workflow expand? | reviewer or adviser unless separately authorised | output quality, process evidence, exceptions, review burden, incidents, missingness | repair, narrow, hold, stop | pre-registered conditions met | | What should leadership receive? | owner of improvement report | decision, evidence, exceptions, displaced work, uncertainty, next gate | N/A | reporting boundary preserves limitations |

A 90-day operating cadence

Days 1–30: diagnose. Select one process, map the complete journey, reconstruct a risk-selected case sample, define the decision unit, identify missing evidence, and compare non-AI alternatives. The output is a signed problem statement, not a pilot announcement.

Days 31–60: design and calibrate. Build representative tasks and a baseline, assign owners, define allowed tools and data, calibrate reviewers, test source and telemetry capture, and write expand, repair, narrow, hold, and stop rules. Run a small dry test to expose protocol failures.

Days 61–90: decide. Execute the bounded test, review cases end to end, compare alternatives, report exceptions and uncertainty, and make the next decision. Preserve adverse or null results. Reopen the agenda if the process, source system, model, prompt, permissions, volume, or consequence changes.

Worked agenda: field-service work-order triage

This synthetic example concerns a 45-person facilities service team that receives work orders by email and portal. The apparent constraint is slow triage. The operations excellence leader owns improvement analysis, the service manager owns response quality, the technology lead owns the platform, and the safety manager can stop the test.

Case reconstruction shows that drafting a triage note is not the main delay. Twenty-two percent of sampled orders lack a site or asset identifier, urgent safety phrases appear in attachments, and technicians use a separate roster to identify qualified responders. The agenda compares four options: require structured intake fields, improve roster integration, use deterministic priority rules, or test AI assistance for extracting details and drafting a routing rationale.

The team implements the structured fields first and measures the new baseline. It then runs a synthetic 30-case AI capability exercise containing a duplicated order, an outdated asset code, a safety warning in a PDF, conflicting urgency labels, and a roster change. Participants must cite source locations, resolve duplicates, verify responder qualification, and escalate safety uncertainty.

Usage is high but not decision evidence. In the first round, several artifacts repeat the email urgency label and miss the PDF warning. Reviewers also disagree about whether an outdated asset code should stop routing. The disposition is repair and retest: clarify the authority hierarchy, add a stop rule for unresolved safety content, calibrate the reviewers, and retain structured intake as the primary process improvement.

The reporting packet states that AI-assisted drafting was tested in a controlled exercise; it does not claim faster live response, improved safety, or readiness for autonomous dispatch. The next gate requires correct safety escalation, stable reviewer interpretation, source traceability, and evidence that review effort does not merely move delay downstream.

Common objections and reporting boundaries

“The AI team owns this” is incomplete: technology ownership does not replace process and outcome accountability. “People already use the tool” shows activity, not capability or process improvement. “The first step is faster” ignores displaced verification, rework, and downstream defects. “A simulation is not the real process” is true; use the simulation for controlled capability evidence and keep live operational conclusions bounded.

Report only what the test supports. Name the workflow, population, configuration, comparison, window, missing data, incidents, reviewer disagreement, and next evidence gate. Do not generalize from one process to organisation-wide capability or from a capability signal to measured business impact.

Two Prune boundary

Two Prune can run realistic work simulations and provide evidence about observed performance in framing, evidence navigation, AI orchestration, verification and risk control, judgment and synthesis, and communication. It does not redesign or operate customer processes, certify controls, establish business impact, assign budgets, or make employment decisions. Customers retain ownership of process design, tools, data, safety, privacy, operational choices, and use of assessment evidence.

Sources

  1. 1.Measuring the success of your service · UK Government
  2. 2.NIST AI RMF Core · National Institute of Standards and Technology
  3. 3.NIST Baldrige: Operations · National Institute of Standards and Technology
  4. 4.GAO AI Accountability Framework · U.S. Government Accountability Office