Practical guide
AI capability pilot design for an enterprise risk leader
An enterprise risk leader should design an AI capability pilot as a bounded decision experiment, not as a miniature rollout or a search for favorable scores. The pilot should reveal whether a specific evidence gap can be reduced safely enough to change an accountable owner’s next decision. Enterprise risk can integrate risks, challenge assumptions, and set evidence gates without taking ownership.
An enterprise risk leader should design an AI capability pilot as a bounded decision experiment, not as a miniature rollout or a search for favorable scores. The pilot should reveal whether a specific evidence gap can be reduced safely enough to change an accountable owner’s next decision. Enterprise risk can integrate risks, challenge assumptions, and set evidence gates without taking ownership away from workflow, learning, privacy, security, finance, or executive decision makers.
Source and role boundary
HM Treasury’s 2026 Magenta Book defines evaluation as a systematic assessment of design, implementation, and outcomes. It emphasizes early evaluation design, explicit questions, proportionate methods, data limitations, learning during delivery, and careful interpretation. The NIST AI Risk Management Framework is voluntary and treats risk tolerance as contextual; it calls for documented roles, intended use, affected parties, monitoring, evaluation, and management of residual risk. These sources support pilot discipline. They do not show that an assessment product will produce business value or give every enterprise risk leader budget authority. HM Treasury: The Magenta Book NIST: AI Risk Management Framework
Sources: [1] [2]
Design the pilot with BOUND
### B — Business decision and bounded claim
Write the decision first. Name its owner, deadline, alternatives, and consequence of uncertainty. A useful question is not “Are people AI fluent?” but “Should the claims-review owner expand, repair, restrict, or stop this assisted workflow after the pilot?”
State the narrow value hypothesis. For example: “A controlled work simulation may reveal recurring source-verification and escalation practices that current training and usage records do not observe.” Keep observed assessment performance separate from workplace transfer, operational outcomes, and financial return.
### O — Owners, affected parties, and operating limits
Create a role map for sponsor, workflow owner, risk integrator, control owners, learning owner, assessment administrator, reviewers, analyst, participant support, and decision maker. Mark each as accountable, responsible, consulted, or informed for every gate.
Define eligible participants, exclusions, voluntariness or required status, accessibility route, data permitted, tools permitted, identity handling, retention, complaint route, and incident response. Do not use the pilot result for an employment decision unless a separately approved process explicitly authorizes that purpose and its safeguards.
### U — Unit of learning and usable sample
Select one representative task, not a broad job label. Specify scenario, evidence set, expected artifact, time boundary, observable practices, and plausible failure modes. The case should create an opportunity to demonstrate source navigation, AI use, verification, judgment, and communication.
Choose a sample to learn about the bounded pilot population, not to manufacture a benchmark. Record invitations, participation, completion, exclusions, technical failures, missing evidence, and reviewer coverage. Describe results as directional when the sample, selection, or reliability evidence does not support broader inference.
### N — Named measures and neutral analysis plan
Predefine implementation measures: reach, completion, accessibility issues, technical failures, evidence availability, reviewer agreement, incidents, and burden. Predefine evidence questions tied to the decision, such as whether participants find the required authoritative sources, challenge unsupported claims, and escalate irreconcilable conflicts.
Specify coding rules, reviewer training, adjudication, missingness, calculation methods, subgroup handling, comparison logic, and prohibited inferences before results are visible. Compare assessment evidence with existing quality, incident, learning, and telemetry records without assuming that any one source is complete.
### D — Decision gate, damage limits, and disposition
Set scale, repair, hold, and exit conditions. A pilot might proceed only if evidence coverage is adequate, the task is interpretable, reviewer disagreements are resolved under the protocol, control incidents are contained, and at least one named decision can change.
Stop when restricted data enters an unapproved system, participant rights cannot be honored, the scenario does not create the intended opportunity, the evidence chain is broken, or the sponsor asks the pilot to support a claim it was not designed to test.
Pilot protocol card
| Field | Required entry | | --- | --- | | decision | owner, deadline, alternatives, consequences | | hypothesis | evidence gap and bounded expected contribution | | task | scenario, sources, artifact, time, failure modes | | population | eligibility, sample rationale, missingness | | roles | workflow, risk, control, learning, review, decision | | measures | implementation, evidence questions, burden, incidents | | analysis | coding, agreement, adjudication, limitations | | gates | scale, repair, hold, exit, next evidence |
Worked pilot: synthetic public-claims review
A communications function uses AI to draft public product claims from approved research. The workflow owner must decide in ten weeks whether to extend the workflow to regional teams. Current evidence shows tool access, training completion, editorial corrections, and three escalated claims, but does not consistently show how authors locate sources, separate evidence from inference, or handle conflicts.
The pilot uses a 35-minute synthetic claims-review task. Each participant receives a product brief, two authoritative reports, one superseded document, a spreadsheet with a denominator change, and an email pressing for a strong launch statement. The required artifact is a claims table with proposed wording, source references, confidence, exclusions, and escalation decisions.
Communications owns the expansion decision. Enterprise risk integrates the risk view and challenges the protocol. Legal reviews claim categories, privacy approves synthetic data, learning owns participant support, and two trained reviewers apply the rubric. Twenty-four volunteers across three regions are invited. The pilot records non-participation and technical failures separately and does not present the group as a workforce norm.
Three questions are preregistered: Do participants reject the superseded source? Do they recompute the denominator before using the spreadsheet claim? Do they escalate a conflict that the evidence cannot resolve? Reviewers also record whether the task created a fair opportunity to demonstrate each behavior and whether evidence is missing.
The scale gate requires usable submissions from the intended regions, no unresolved high-severity incident, acceptable evidence coverage under the preselected rule, reviewer disagreements adjudicated, and a concrete workflow or support action. Results show repeated denominator errors but strong source-version checking. The workflow owner chooses a bounded repair: add a calculation check and claims ledger, then sample live editorial corrections for six weeks. The pilot does not establish that the repair will improve business outcomes.
Report without overclaiming
Report participation, implementation quality, task-bounded findings, missing evidence, reviewer agreement, exceptions, participant burden, and the decision taken. Separate observed performance from explanations and recommendations. Do not publish a precise benchmark, causal training effect, compliance conclusion, or return claim that the design cannot support.
Two Prune provides realistic AI-enabled work assessments and evidence about observed participant performance. It may supply one controlled evidence layer in this pilot. It does not set risk appetite, certify capability, predict workplace performance, prove programme impact, or make employment decisions.
Sources
HM Treasury: The Magenta Book
- NIST: AI Risk Management Framework
- NIST: AI Risk Management Framework 1.0
Sources: [1] [2] [3]
Sources
- 1.HM Treasury: The Magenta Book · UK Government
- 2.NIST: AI Risk Management Framework · National Institute of Standards and Technology
- 3.NIST: AI Risk Management Framework 1.0 · National Institute of Standards and Technology