Practical guide

AI capability pilot design for a Chief Learning Officer

A Chief Learning Officer should design an AI capability pilot to rehearse one real learning decision, not to produce a miniature organization-wide rollout. The pilot needs a bounded population, realistic task, observable evidence, comparison logic, review method, safeguards, and pre-registered exit conditions. Its result should distinguish what changed in the pilot from what remains unknown.

By Two Prune

A Chief Learning Officer should design an AI capability pilot to rehearse one real learning decision, not to produce a miniature organization-wide rollout. The pilot needs a bounded population, realistic task, observable evidence, comparison logic, review method, safeguards, and pre-registered exit conditions. Its result should distinguish what changed in the pilot from what remains unknown about workplace transfer or business outcomes.

Start with one decision and role map

Write the decision before the task. Examples include whether to revise a learning pathway, add verification practice, investigate a non-learning cause, or test transfer with another group.

Classify the CLO as owner, influencer, reviewer, or collaborator for that decision, then name the actual operational owner, privacy and accessibility reviewers, information-security reviewer, managers or subject experts, and final decision maker. Do not infer budget or employment authority from the title.

The U.S. Office of Personnel Management notes that training is not always the best or only response to a performance gap. CDC recommends defining an evaluation’s purpose, users, questions, and data methods early, and its Quality Training Standards distinguish needs, learning objectives, evaluation, learner assessment, and follow-up support. These sources support a decision-led pilot; they do not establish that a particular assessment or programme causes workplace outcomes. OPM: Planning and evaluating training CDC: Building a training evaluation plan CDC: Quality Training Standards

Sources: [1] [2] [3]

Use the PILOT protocol

### P — Purpose, population, and alternatives

Define the decision, date, population, inclusion logic, and alternatives. Compare at least maintain, revise learning, add a task-based assessment, change workflow or support, and investigate another cause. State which evidence would distinguish them.

Recruitment and participation status must be visible. A convenience sample may be appropriate for feasibility, but it cannot support a population-wide conclusion. Record invitations, starts, completions, withdrawals, missing evidence, and accommodations separately.

### I — Instrument and realistic task

Design a task that resembles the target work without copying confidential production material. Specify:

The task should elicit framing, evidence navigation, AI orchestration, verification, judgment, and communication where relevant. Do not add difficulty that does not serve the pilot decision.

### L — Logic, measures, and learning comparison

Create an evidence map before collection:

| Layer | Pilot question | Measure | Interpretation boundary | | --- | --- | --- | --- | | Reach | Who could participate? | invitations, access, accommodations | not capability | | Participation | Who attempted and completed? | starts, completion, time | not learning | | Learning | What changed in the learning context? | pre-specified knowledge or practice evidence | not transfer | | Task performance | What was demonstrated in the simulation? | rubric evidence from process and deliverable | bounded to task and conditions | | Transfer | Is the behaviour used in work? | later sampled work evidence | requires separate design | | Outcome | Did an operational measure change? | quality, rework, cycle time, incidents | attribution may remain unresolved |

If using a pre/post or comparison design, explain what is held constant, what can contaminate the comparison, and what the design cannot attribute. Avoid creating a composite score that hides the failure pattern the learning team needs.

### O — Operations, oversight, and objections

Pilot the operating process as well as the task. Measure setup effort, participant support, reviewer time, disagreement, adjudication, technical failures, privacy exceptions, accessibility issues, and reporting burden.

Pre-answer common objections:

### T — Thresholds, timeline, and terminal decision

Pre-register conditions for:

Assign every exit an owner, evidence packet, and next date. Do not invent a positive threshold after seeing the results.

  • assignment, audience, materials, tools, and time boundary;
  • ambiguity, tradeoffs, and evidence traps relevant to the learning objective;
  • expected deliverable and acceptance criteria;
  • process evidence needed to interpret the output;
  • AI use allowed, restricted, or required;
  • accessibility and support arrangements;
  • synthetic or anonymized data controls.
  • “Completion already tells us enough.” State the distinct decision that requires performance evidence.
  • “The scenario is artificial.” Document which work conditions are represented and which require later transfer evidence.
  • “Managers can observe the same thing.” Compare structured task evidence with a defined manager-observation method rather than dismissing either source.
  • “Individual results may be misused.” Fix purpose, access, retention, communication, human review, and prohibited uses before collection.
  • “A pilot cannot prove ROI.” Agree, then define the feasibility and decision-usefulness assumptions it can test.
  • scale the next test: evidence is decision-useful, review is workable, safeguards function, and results justify broader sampling;
  • revise: task, rubric, learning design, tooling, or support has a repairable failure;
  • investigate: evidence points to a non-learning cause or unresolved alternative;
  • hold: required authority, accessibility, privacy, source, or interpretation evidence is missing;
  • stop: the pilot duplicates existing evidence, cannot support the decision, or creates disproportionate burden or risk.

Worked pilot: verification after analyst training

A synthetic organization has high completion and satisfaction for an AI-analysis course, yet reviewers continue to find unsupported claims and spreadsheet errors. The CLO needs to decide whether the next learning cycle should add verification practice.

The pilot invites 36 volunteers from two analyst teams. Participants complete a controlled decision task containing a plausible unsupported citation, a denominator error, and conflicting source dates. The task records opened sources, calculations, AI interactions, revisions, and the final deliverable. It excludes unrelated personal data and does not automate an employment decision.

The current knowledge check is retained as a comparison measure, not a control group that proves causation. Two trained reviewers independently score the first sample, adjudicate disagreements, and record rubric changes. The pilot also measures review time, accommodation use, missing evidence, and technical support.

The synthetic result shows that most participants question the citation, fewer recalculate the denominator, and still fewer explain the consequence for the recommendation. One rubric item produces repeated disagreement because it combines calculation accuracy and communication. The learning team reaches revise: split the rubric item, add calculation-verification practice, and repeat the task with a more representative sample before considering expansion.

The report does not claim workplace transfer or operational impact. It records an alternative explanation—unfamiliar spreadsheet structure—and assigns a usability check. The evidence packet contains the recruitment flow, task version, source set, reviewer agreement, adjudication record, failure taxonomy, operating burden, safeguards, exit decision, and next test.

Pilot brief template

| Field | Required entry | | --- | --- | | Decision and owner | exact choice, date, accountable person | | Population and recruitment | inclusion rule, invitation, participation states | | Task and learning objective | realistic work, materials, tools, deliverable, constraints | | Evidence and measures | process, output, comparison, missingness, limitations | | Review method | rubric, reviewer preparation, independence, adjudication | | Safeguards | purpose, access, retention, accessibility, participant communication | | Operating measures | setup, support, reviewer time, failures, exceptions | | Exit conditions | scale, revise, investigate, hold, stop | | Next evidence gate | owner, method, date, unresolved question |

Two Prune provides realistic AI-enabled work assessments and evidence about observed task performance. It does not deliver a learning programme, establish that training caused an outcome, or make employment decisions. The customer remains responsible for pilot purpose, governance, interpretation, and follow-up.

Sources

U.S. Office of Personnel Management: Planning and evaluating training

  • Centers for Disease Control and Prevention: Building a training evaluation plan
  • Centers for Disease Control and Prevention: Quality Training Standards

Sources: [1] [2] [3]

Sources

  1. 1.OPM: Planning and evaluating training · U.S. Office of Personnel Management
  2. 2.CDC: Building a training evaluation plan · Centers for Disease Control and Prevention
  3. 3.CDC: Quality Training Standards · Centers for Disease Control and Prevention