Practical guide

How an L&D leader can design an applied AI capability pilot

An applied AI capability pilot should test whether a bounded assessment process produces useful, interpretable evidence for a named learning decision. It should not be a small rollout with vague success language. The learning and development leader may own programme design, influence investment, review evidence, or coordinate the trial; local governance determines who approves technology,.

By Two Prune

An applied AI capability pilot should test whether a bounded assessment process produces useful, interpretable evidence for a named learning decision. It should not be a small rollout with vague success language. The learning and development leader may own programme design, influence investment, review evidence, or coordinate the trial; local governance determines who approves technology, privacy, accessibility, work standards, and any later expansion. Use PILOT to design the trial: pin the decision, include representative people and conditions, lock the evidence plan, operate the process safely, and trigger a preset decision. The output is a pilot protocol that can be reviewed before any participant is invited.

P: Pin one decision and the leader's authority

Write the decision the pilot will inform. Examples include whether to revise a learning module, add realistic practice, retain manager observation, or test a larger cohort. Avoid goals such as "prove AI readiness" or "validate the platform" because they do not identify a bounded choice or falsifiable evidence need.

Record the accountable decision owner and classify the L&D leader as owner, influencer, reviewer, or collaborator for each part of the pilot. A functional leader should approve whether the task resembles relevant work. Accessibility, privacy, security, technology, and workforce representatives retain their own responsibilities. Finance or an executive sponsor may own investment approval.

The protocol should name:

  • the decision and date;
  • the target work behaviour and why it matters;
  • current evidence and its limitations;
  • credible alternatives, including no change;
  • the owner of the final decision;
  • exclusions, especially employment decisions and general workforce surveillance.

I: Include representative people, tasks, and conditions

Define the population before choosing the sample. Describe roles, experience, starting exposure to AI, work context, language, accessibility needs, location, and any operational constraints that could affect participation or interpretation.

Select a sample that gives the pilot a fair opportunity to reveal process problems and varied experiences. A convenience group of enthusiasts can test mechanics, but it cannot represent everyone affected. State the limitation rather than applying a cosmetic diversity label.

Use synthetic or explicitly approved non-sensitive materials. Include realistic ambiguity, evidence conflicts, and verification opportunities without importing live customer, personnel, classified, or controlled data. Run accessibility and technical checks before launch and provide an equivalent route where a feature creates an avoidable barrier.

The 2026 UK employer guide on AI upskilling recommends practical, reachable, integrated, modular, expandable, and sustainable learning. Those principles support task relevance and inclusion, but they do not establish that a particular pilot will produce learning transfer or business results.

L: Lock the evidence plan before the pilot starts

Separate process acceptance from capability evidence and later outcomes.

Process acceptance can cover task access, completion, evidence capture, reviewer workload, accessibility, incidents, and whether the report explains its basis. Capability evidence can cover task framing, source navigation, AI use, verification, judgment, and communication within the simulation. Later outcomes, such as workplace quality or productivity, require separate designs and should not be inferred from an assessment alone.

Create a measure register with the construct, observation opportunity, source, missingness rule, reviewer, interpretation limit, and decision use. Lock the rubric, task version, time window, review process, and change procedure. If the protocol changes, keep the original and explain the effect on comparability.

The UK guidance on impact evaluation of AI interventions recommends planning evaluation early, tailoring it to context and risk, and being explicit about objectives. A small operational pilot may not support causal impact claims. Its design should match the decision it can genuinely inform.

O: Operate the pilot with visible safeguards

Rehearse the complete journey with synthetic records: invitation, briefing, task access, support, submission, evidence capture, review, reporting, correction, and deletion or retention. Assign an incident owner and a pause route.

During operation, log departures from the protocol. Do not repair a participant's missing record in a way that hides the failure. Mark it missing, investigate the cause, and decide whether the record remains interpretable. Keep participant support separate from scoring judgment where practical.

Monitor for:

  • access or accommodation failures;
  • task ambiguity that prevents the intended behaviour from being observed;
  • unavailable or contradictory source materials;
  • evidence-capture gaps;
  • reviewer disagreement or unsupported inferences;
  • technical failures, inappropriate data exposure, or unexpected distress;
  • pressure to reuse results beyond the agreed purpose.

T: Trigger a preset decision, not a retrospective success story

Define thresholds before results are visible. Thresholds should reflect the pilot's purpose, not manufacture a pass rate. Possible outcomes are stop, repair and rerun, continue within the same boundary, or propose a separately reviewed expansion.

Use a decision table with process criteria, evidence criteria, stop conditions, owner, and action. Report denominators and missingness. A high completion rate cannot cancel an accessibility failure, and a useful report cannot compensate for a task that did not create the intended evidence opportunity.

Pilot protocol artifact

| Field | Required entry | |---|---| | Decision | One choice, accountable owner, and decision date | | Population | Who is in scope, exclusions, and representation limits | | Task | Work context, materials, time, deliverable, and evidence opportunities | | Alternatives | Current method and credible non-pilot options | | Measures | Construct, source, missingness, reviewer, and interpretation limit | | Acceptance | Accessibility, technical, evidence, and reviewer criteria | | Safeguards | Data boundary, support, incident, pause, retention, and access rules | | Stop rules | Conditions that end or reset the pilot | | Outcomes | Stop, repair, continue, or seek separate expansion approval |

Worked example: a verification-practice pilot

A synthetic organisation wants to decide whether realistic practice adds useful evidence after introductory AI training for policy analysts. L&D coordinates the pilot, the policy director approves the work scenario, privacy and accessibility reviewers approve the protocol, and the transformation sponsor owns the later programme decision.

Sixteen volunteers with varied AI experience complete a 40-minute synthetic briefing task. The task includes a current policy source, an outdated summary, a spreadsheet with one denominator error, and an AI drafting tool. The evidence plan looks for source selection, calculation checking, uncertainty handling, and correction of the final brief. It does not treat prompt count or speed as capability.

Preset stop rules include any material accessibility defect, exposure of non-synthetic information, more than two unexplained evidence-capture failures, or reviewer inability to trace a rating to observed evidence. Expansion is considered only if the task creates the intended evidence opportunities, reports remain interpretable, reviewers can explain disagreements, and participants can use the process without unresolved barriers.

The pilot finds that two source-open events are missing because of a viewer defect. The team pauses reporting, repairs the viewer, and reruns synthetic acceptance tests. It does not impute those events or declare the pilot successful from the remaining records. The final decision concerns whether the assessment process is ready for another bounded test, not whether the workforce is capable.

Reporting boundaries for Two Prune

Two Prune can provide a realistic simulation, capture available process and deliverable evidence, and report observed assessment performance with confidence and calibration limits. It does not prove learning transfer, productivity, return on investment, broad role readiness, or later workplace performance. It does not make employment decisions. Early comparison groups should be described as directional or unavailable rather than stable benchmarks.

Primary sources

UK Government, Skills for AI: What works for AI upskilling in the UK

  • UK Government, Guidance on the Impact Evaluation of AI Interventions
  • UK Government, Evaluation of the Flexible AI Upskilling Fund

Sources: [1] [2] [3]