Practical guide

How a Chief Learning Officer can build the case for applied AI capability evidence

A Chief Learning Officer’s business case should connect a specific learning decision to evidence that existing measures cannot supply. It should compare credible alternatives, make assumptions visible, and fund the smallest pilot that can change the decision. Course completion, participant sentiment, task performance, workplace transfer, and business outcomes belong in separate evidence layers.

By Two Prune

A Chief Learning Officer’s business case should connect a specific learning decision to evidence that existing measures cannot supply. It should compare credible alternatives, make assumptions visible, and fund the smallest pilot that can change the decision. Course completion, participant sentiment, task performance, workplace transfer, and business outcomes belong in separate evidence layers.

Define the case as a choice

Write a one-sentence decision with alternatives and a date. For example: “Should we add a realistic AI-verification assessment to the analyst learning pathway, improve the current knowledge check, or make no change this quarter?”

Then classify the CLO’s role for this decision:

Do not infer budget authority, employment authority, or ownership of operational outcomes from the title. Name the actual decision maker and every required reviewer.

The U.S. Office of Personnel Management advises beginning with a needs assessment because training is not always the solution and is rarely the only one. Its evaluation field guide is intended to connect training evidence with stakeholder decisions. CDC’s current training-evaluation guidance recommends defining the purpose, users, questions, and data methods early, while distinguishing process questions from outcomes such as learning and transfer. These public-sector sources offer useful evaluation discipline; they do not determine a private organization’s investment case. OPM: Planning and evaluating training CDC: Building a training evaluation plan

  • owner of the learning architecture or evaluation plan;
  • influencer on investment, workforce strategy, or technology adoption;
  • reviewer of evidence quality, accessibility, and development use;
  • collaborator with operational, data, security, privacy, legal, finance, or people teams.

Sources: [1] [2]

Build the CASE dossier

CASE keeps the proposal tied to a decision rather than a generic claim that more measurement is better.

### C — Counterfactual and alternatives

Describe the observed performance gap and the evidence for it. List plausible non-learning causes such as unclear expectations, weak source access, unsuitable tooling, incentives, workload, or manager reinforcement. Compare at least three actions:

For each option, state what it can reveal, what it cannot reveal, the operating burden, privacy and accessibility implications, and the decision it enables.

### A — Assumptions and value pathway

Map the pathway without skipping evidence layers:

access → participation → learning → bounded task performance → workplace transfer → operational outcome

Mark which links are currently observed, which the pilot will test, and which remain hypotheses. If the proposal uses financial language, separate measured cost from estimated cost, and separate avoided rework or time from cashable savings. Give every estimate an owner, range, source, and expiry date.

The business case may value faster diagnosis, clearer development priorities, reduced review burden, or better learning design even before business outcomes are measurable. Do not turn those plausible benefits into achieved results.

### S — Smallest decision-changing study

Design a pilot around one role-like work context and one decision. Specify:

Avoid requiring the pilot to prove organization-wide impact. Its job is to expose whether the proposed evidence changes a real learning decision and whether the collection and review process is workable.

### E — Evidence report and exit decision

The final dossier should show distributions and failure patterns, not just an average score. Separate:

Close with one disposition: scale the next test, revise the intervention, investigate another cause, hold for missing evidence, or stop. Assign an owner and date for the next evidence gate.

  • keep the current programme and measurement;
  • improve the existing learning or knowledge check;
  • add a realistic task-based assessment;
  • change workflow, tooling, support, or management practice;
  • investigate before intervening.
  • population, inclusion rule, invitation, and participation status;
  • realistic task, materials, time boundary, and expected deliverable;
  • process and output evidence to capture;
  • scoring criteria, reviewer preparation, and quality checks;
  • current measure or baseline used for comparison;
  • missing-data, accommodation, withdrawal, and appeal handling;
  • cost, staff time, technical dependencies, and security review;
  • pre-registered scale, revise, investigate, hold, and stop conditions.
  • reach and completion;
  • knowledge or confidence measures;
  • performance on the bounded task;
  • evidence strength and scoring uncertainty;
  • operational burden and reviewer agreement;
  • transfer or business outcomes, only when directly observed under an appropriate design.

Copyable decision map

| Decision question | Option | Evidence already available | Critical unknown | Pilot measure | Cost or burden | Risk / safeguard | Exit condition | Owner | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | exact learning decision | maintain / improve / assess / change workflow / investigate | source and date | uncertainty that changes choice | bounded observable evidence | measured or estimated, labelled | privacy, accessibility, security, interpretation | scale / revise / investigate / hold / stop | named person |

Keep this map to one page. Put detailed methods, source definitions, and approvals in appendices so decision makers can inspect them without losing the choice being made.

Worked business case: verification practice for analysts

A synthetic organization sees strong completion and satisfaction for its AI-analysis course, but quality reviewers still report unsupported claims and spreadsheet errors. The CLO is asked to fund a full assessment programme.

The CASE dossier begins with three alternatives: add another knowledge module, introduce manager observation, or run a small realistic assessment before changing the curriculum. Existing evidence confirms course participation and shows a pattern in reviewer issue logs, but the logs do not identify who used AI, whether expectations were clear, or whether the errors reflect knowledge, workflow, or time pressure.

The proposed pilot invites 36 volunteers from two analyst teams. They complete a synthetic decision task containing a plausible unsupported citation, a denominator error, and conflicting source dates. The pilot records source review, calculations, revisions, final deliverable, reviewer time, and scoring questions. Results are reported for the pilot only and are not used to automate an employment decision.

The cost table distinguishes known facilitator and reviewer hours from estimated platform and participant time. It does not monetize hypothetical future error avoidance. The scale condition requires that the task reveal actionable breakdowns, reviewers reach the pre-specified agreement floor after adjudication, accommodations work, and the learning team can name a concrete programme change. A stop condition applies if the evidence duplicates the knowledge check or the review burden is impractical.

Suppose the synthetic result shows most participants identify the citation issue, while fewer recalculate the denominator and many fail to explain the decision consequence. Reviewers agree on the evidence after one rubric clarification. The exit decision is revise, not “roll out everywhere”: add calculation-verification practice, repair the ambiguous rubric item, and repeat the bounded challenge with a more representative group. Workplace transfer and operational effect remain later evidence gates.

Address predictable objections

“We already have completion data.” Completion describes participation. Show exactly which learning or task-performance decision requires additional evidence.

“Managers can observe capability.” Manager evidence may add real workplace context, but opportunity, task difficulty, expectations, and observation quality vary. Define a structured observation method and compare it with the pilot rather than treating either source as complete.

“A pilot will not prove return on investment.” Agree. Use the pilot to test decision usefulness, feasibility, evidence quality, and the value pathway’s assumptions. Reserve outcome or financial claims for a design that can support them.

“Individual results could be misused.” Specify purpose, access, retention, participant communication, human review, and prohibited uses before collection. Report programme findings at an appropriate level and preserve uncertainty.

“The scenario is too artificial.” Connect materials, constraints, and acceptance criteria to real work while keeping the task controlled enough to interpret. Document what the simulation omits and add transfer evidence later if the decision needs it.

Two Prune provides realistic AI-enabled work assessments and evidence about how a participant frames a task, navigates sources, uses AI, verifies output, exercises judgment, and communicates a deliverable. It does not deliver the learning programme, establish that training caused an outcome, or make employment decisions. The CLO’s organization remains responsible for purpose, governance, interpretation, and action.

Sources

U.S. Office of Personnel Management: Planning and evaluating training

  • Centers for Disease Control and Prevention: Building a training evaluation plan

Sources: [1] [2]

Sources

  1. 1.OPM: Planning and evaluating training · U.S. Office of Personnel Management
  2. 2.CDC: Building a training evaluation plan · Centers for Disease Control and Prevention