Practical guide

AI capability pilot design for a digital transformation leader

A digital transformation leader should design an AI capability pilot as a bounded decision experiment, not a showcase or procurement shortcut. The pilot should test whether a defined evidence method helps an accountable owner make a specific transformation decision. It needs credible alternatives, representative work, pre-set thresholds, operational safeguards, reporting limits, and a breakpoint.

By Two Prune

A digital transformation leader should design an AI capability pilot as a bounded decision experiment, not a showcase or procurement shortcut. The pilot should test whether a defined evidence method helps an accountable owner make a specific transformation decision. It needs credible alternatives, representative work, pre-set thresholds, operational safeguards, reporting limits, and a breakpoint that allows the organization to repair, extend, scale, or stop.

Authority and source boundary

Classify the transformation leader as owner, influencer, reviewer, or collaborator for each decision. Service owners, learning leaders, privacy, security, accessibility, finance, procurement, and workforce representatives may hold distinct authority. A transformation title does not imply ownership of every budget or release.

The UK Government Digital, Data and Technology Playbook recommends outcome-focused, proportionate testing, early pilot planning, measurable success criteria, representative design, governance, knowledge transfer, and explicit breakpoints before scale. The Department for Business and Trade’s digital evaluation strategy treats monitoring and evaluation as part of digital delivery rather than an afterthought. These public-sector sources inform the method; they do not prove that a Two Prune assessment or any pilot will create business impact. UK Government: Digital, Data and Technology Playbook Department for Business and Trade: Digital evaluation and performance analysis strategy

Sources: [1] [2]

Design the pilot with PILOT

### P — Problem, decision, and authority

State the operational or learning decision, decision date, affected users, current process, accountable owner, and why existing evidence is insufficient. Separate capability questions from adoption, access, satisfaction, productivity, and service outcomes.

Write the decision options before designing the exercise: no change, improve guidance, repair the workflow, add manager observation, review work samples, run a realistic assessment, combine methods, investigate another cause, or stop.

Create an authority map:

| Decision | Owner | Influencers | Required reviewers | Evidence needed | Latest date | | --- | --- | --- | --- | --- | --- | | exact pilot or scale choice | named role | relevant functions | control owners | bounded packet | date |

### I — Indicators, alternatives, and inference

Define what the pilot can observe. For an AI-enabled work assessment, signals might include task framing, source navigation, AI interaction, verification, judgment, revisions, and final communication. State what each indicator does not establish.

Compare the proposed pilot with current work evidence, usage logs, training completion, surveys, manager observation, operational metrics, and no additional measurement. Document additionality: what decision-relevant uncertainty does the pilot reduce that cheaper evidence cannot?

Use a simple logic chain, but label it as a hypothesis. Observed performance in a realistic task may inform a capability decision. It does not by itself establish workplace transfer, productivity, ROI, or service improvement.

### L — Limits, cohort, and learning design

Define participant population, invitation rule, sample rationale, task family, time limit, accessibility support, language, prior tool access, reviewer design, missingness, and exclusion rules. Avoid selecting only enthusiasts or only people already successful with the tool.

Use representative ambiguity and evidence traps without introducing irrelevant difficulty. Freeze the brief, source set, tool configuration, rubric, and material version for the cohort. Predefine how incomplete sessions, technical failures, accommodations, and suspected evidence-capture gaps will be handled.

The first pilot can test feasibility and decision usefulness. If workplace transfer matters, define a later and separate evidence gate using approved sampled work or service evidence.

### O — Operations, safeguards, and ownership

Map the end-to-end operating path: invitation, consent or notice where applicable, access, support, incident handling, evidence capture, submission, scoring, adjudication, reporting, retention, deletion, and participant questions.

Assign owners and fallbacks. Run a dry test covering accessibility, browser or desktop requirements, timing, source visibility, AI tool behavior, evidence logging, reviewer access, and report generation. Do not interpret missing telemetry as poor performance.

Set data minimization, approved synthetic or anonymized materials, role-based access, retention, deletion, and escalation before launch. Route jurisdiction-specific privacy, employment, labor, accessibility, or procurement questions to qualified owners.

### T — Thresholds, breakpoint, and truthful reporting

Pre-register thresholds for technical completion, evidence completeness, reviewer agreement, adjudication burden, accessibility, participant support, consequential error patterns, and decision usefulness. Include mandatory holds that cannot be averaged into a success score.

At the breakpoint, choose proceed, limited extension, repair and rerun, use another method, hold, or stop. Scale only when the evidence supports the next stage and operating capacity exists.

Report observed performance, evidence strength, missingness, reviewer variation, burden, limitations, and residual uncertainty. Do not turn a small pilot percentile into a mature benchmark or use the report as an automated employment decision.

Worked pilot: internal procurement briefs

A synthetic organization wants to know whether 60 programme staff can use AI to prepare procurement option briefs without losing source traceability. Tool activation is high, but leaders cannot tell whether staff verify commercial claims or distinguish requirements from vendor assertions.

The transformation leader is the pilot owner. Procurement owns the option-brief standard; security and privacy review the environment; accessibility reviews the task; learning collaborates on follow-up; service owners decide any workflow change. The scale decision remains with the programme board.

Alternatives include a source-citation clinic, manager review of live briefs, a checklist, a realistic assessment, or a combined checklist plus assessment. Live work samples contain sensitive supplier information and differ widely, so the board chooses a synthetic assessment for the first evidence layer and retains a later sampled-work gate.

The pilot uses 18 synthetic participants across three programme teams. The task asks each person to compare two delivery options using an inbox message, requirement note, cost table, risk log, and two deliberately inconsistent supplier statements. Participants may use an approved AI assistant. The evidence record captures opened sources, AI interactions, calculations, revisions, verification steps, escalation, and the final brief.

The rubric covers the six Two Prune assessment pillars. Reviewers also record three pilot-specific events: unsupported supplier claim accepted, cost inconsistency detected, and uncertainty escalated. Two reviewers independently score six submissions; disagreements receive adjudication. A technical failure is coded as missing evidence, not low capability.

Pre-set gates require at least 16 technically complete sessions, no unresolved accessibility blocker, complete source and submission records, acceptable reviewer agreement on the double-scored sample, explainable disagreement, support burden within the staffing plan, and a report that separates observed task performance from transfer.

The synthetic result is mixed: 17 sessions complete, reviewers agree on broad evidence patterns, but four participants cannot access a spreadsheet viewer at 200% zoom and two event logs omit source-close events. The breakpoint decision is repair accessibility and evidence capture, then rerun the affected cases. The team does not scale or characterize the workforce as ready or unready.

The evidence packet includes the decision charter, authority map, alternatives, task and source versions, sample rationale, dry-run record, accessibility review, data map, rubric, reviewer calibration, issue log, results, breakpoint decision, and next evidence gate.

Objections and reporting boundaries

“A simulation is not real work.” Correct. It provides standardized task evidence. If transfer is the decision, add a later approved workplace-evidence design.

“A pilot this small cannot benchmark the organization.” Correct. Use descriptive patterns and uncertainty; do not present a mature percentile.

“Managers already know their teams.” Manager evidence may be useful but can vary in opportunity, observation, task difficulty, and documentation. Compare it rather than dismissing it.

“Successful completion proves the programme works.” It does not. Completion, observed capability, transfer, service impact, and financial return are different constructs.

Two Prune provides realistic AI-enabled work assessments, scores observed performance, and explains supporting evidence. It does not own the wider transformation, validate business impact, establish capability transfer, or make employment decisions.

Sources

UK Government: Digital, Data and Technology Playbook

  • Department for Business and Trade: Digital evaluation and performance analysis strategy

Sources: [1] [2]