Practical guide

AI capability measurement guide for the Chief Learning Officer

A Chief Learning Officer considering AI capability measurement needs a decision agenda, not another activity dashboard. Course access, completion, satisfaction, knowledge checks, performance in a realistic task, transfer to work, and business outcomes answer different questions. Treating them as interchangeable produces confident reporting without a usable decision. This guide uses the LEARN.

By Two Prune

A Chief Learning Officer considering AI capability measurement needs a decision agenda, not another activity dashboard. Course access, completion, satisfaction, knowledge checks, performance in a realistic task, transfer to work, and business outcomes answer different questions. Treating them as interchangeable produces confident reporting without a usable decision. This guide uses the LEARN agenda to define the decision, map evidence, design a bounded pilot, and report what remains unknown. The CLO is normally the learning-system owner or a major influencer; budget, employment, legal, security, and operational authority may sit elsewhere. Confirm those roles for the organization instead of assuming them from the title.

L — Locate the decision and the performance gap

Start with one decision and a date. Examples include whether to revise a programme, add practice for a weak capability, extend a pilot to another team, or commission a deeper transfer study. Do not begin with “measure AI fluency” as an unbounded objective.

Record:

The U.S. Office of Personnel Management notes that training is not always the best solution and is rarely the only one; expectations, work environment, and job fit may also affect performance. Its training-evaluation guidance therefore supports beginning with needs and the intended decision rather than presuming that training caused or will repair the gap. OPM: Planning and evaluating training

  • the work outcome that matters and who owns it;
  • the observed gap, its evidence, and plausible non-learning causes;
  • the learner population, workflow, tools, and constraints;
  • the intervention being considered and credible alternatives;
  • the decision maker, reviewers, evidence deadline, and reversibility of the decision.

Sources: [1]

E — Establish the evidence ladder

Build a decision-and-evidence map before selecting measures.

| Evidence level | Question answered | Example measure | Common overreach | | --- | --- | --- | --- | | Access | Could the person reach the learning? | enrolment, login, accommodation delivered | calling access capability | | Participation | Did they engage with the intervention? | attendance, completion, practice attempts | calling completion learning | | Learning | Did knowledge or skill change in the learning context? | scored exercise, pre/post knowledge task | calling a knowledge gain work transfer | | Task performance | Can they produce and verify useful work in a bounded simulation? | evidence-backed assessment task | calling one task organization-wide capability | | Transfer | Is the capability applied in real work under relevant conditions? | sampled work review, manager observation, workflow evidence | attributing change to training without alternatives | | Outcome | Did the operational result change? | quality, rework, cycle time, incidents | claiming causation from a coincident trend |

The CDC Quality Training Standards distinguish learning objectives, training evaluation, learner assessment, and follow-up support. CDC’s evaluation guidance also separates process questions from outcome questions and recommends defining the evaluation’s purpose, users, questions, and data methods early. These distinctions support an evidence ladder but do not prescribe a particular company’s learning strategy. CDC: Quality Training Standards CDC: Building a training evaluation plan

Sources: [2] [3]

A — Assign roles, safeguards, and alternatives

Create a role map for the exact decision:

Name the participant-data purpose, minimum necessary fields, access, retention, appeal or correction route, and reporting groups. Keep assessment evidence separate from an automated employment recommendation. Decide whether individual results are for development, programme evaluation, another stated purpose, or some combination that has been explicitly reviewed.

Compare at least three plausible actions: maintain the current programme, improve targeted practice, run a task-based assessment, change workflow or tooling, or investigate a non-learning cause. The evidence map should make clear what would distinguish those alternatives.

  • Owner: accountable for the learning decision and follow-through.
  • Influencer: shapes objectives, population, programme, or interpretation.
  • Reviewer: checks evidence quality, privacy, accessibility, security, legal, or methodological boundaries.
  • Collaborator: supplies operational context, data, facilitation, or participant support.

R — Run a bounded decision pilot

A useful pilot rehearses the future decision on a small, defined population. Specify:

Pre-register the interpretation. For example, improved performance on the pilot task may support revising the programme or testing transfer; it does not by itself establish sustained workplace impact. Completion data may reveal reach problems but cannot diagnose verification skill.

  • the population and inclusion logic;
  • the learning objective and realistic task;
  • the evidence captured from process and deliverable;
  • the scoring or review rule and assessor preparation;
  • the comparison or baseline, if one is defensible;
  • accessibility and support arrangements;
  • missing-data, withdrawal, and exception handling;
  • scale, revise, investigate, and stop conditions.

N — Narrate the decision, not just the scores

The CLO’s report should contain:

Avoid a single composite that hides whether the issue is access, knowledge, task execution, or transfer. When the sample is small or unrepresentative, label the result directional. Do not generalize beyond the population and task observed.

  • decision and accountable owner;
  • population, participation, and missingness;
  • measures at each evidence level actually observed;
  • distribution and uncertainty, not only an average;
  • capability strengths and breakdowns tied to task evidence;
  • differences that require investigation rather than causal stories;
  • operational burden, accommodations, and review exceptions;
  • recommended action, alternative, owner, date, and next evidence gate.

Worked agenda: verification practice after AI training

A synthetic organization has delivered a two-hour course on reviewing AI-generated analysis. The CLO is asked whether to extend it to all analysts. Ninety-two per cent of the pilot group completed the course, and satisfaction averaged 4.6 out of 5. Those results show participation and reaction, not whether analysts can detect unsupported claims.

Using LEARN, the team defines the decision as: “Should the next learning cycle add evidence-verification practice before expansion?” The operational partner owns the quality requirement, the CLO owns the programme decision, the privacy and accessibility leads review the evidence process, and team managers collaborate on realistic conditions.

Forty volunteers complete a bounded simulation in which an AI draft contains a plausible but unsupported market claim and a spreadsheet with a denominator error. The evidence packet records which sources they open, whether they recalculate the metric, their revisions, and the final deliverable. It does not collect unrelated personal information.

The synthetic results are deliberately mixed: 31 participants identify the unsupported claim, 19 catch the denominator error, and 14 both correct the calculation and explain the consequence. The small volunteer sample prevents a workforce-wide conclusion. Still, the breakdown answers the pilot decision: the next cycle should include calculation-verification practice and a repeat challenge before wider rollout. The team holds any claim about workplace transfer until sampled real-work evidence and relevant operational measures are available.

The report therefore separates completion, task performance, and the next evidence gate. It also records an alternative explanation—participants may be unfamiliar with the spreadsheet format—and assigns a usability check before changing the curriculum.

Questions the CLO should ask vendors and internal teams

Two Prune provides realistic AI-enabled work assessments and evidence about how a participant frames a task, navigates sources, uses AI, verifies output, exercises judgment, and communicates a deliverable. It does not deliver a learning programme, prove that training caused an outcome, or make employment decisions. A customer should align any assessment use with its stated purpose, governance, and human review process.

  • What exact decision will this evidence change?
  • Which levels of the evidence ladder are measured, and which are not?
  • Can a reviewer inspect the source, method, scoring rule, and exceptions?
  • How are realistic tasks kept comparable without becoming generic quizzes?
  • How are accessibility, privacy, missing evidence, and participant questions handled?
  • What is the smallest pilot that can expose a wrong assumption?
  • Which result would lead us to revise, investigate, hold, or stop?
  • What additional evidence is needed before discussing transfer or business outcomes?

Sources

U.S. Office of Personnel Management: Planning and evaluating training

  • Centers for Disease Control and Prevention: Quality Training Standards
  • Centers for Disease Control and Prevention: Building a training evaluation plan

Sources: [1] [2] [3]

Sources

  1. 1.OPM: Planning and evaluating training · U.S. Office of Personnel Management
  2. 2.CDC: Quality Training Standards · Centers for Disease Control and Prevention
  3. 3.CDC: Building a training evaluation plan · Centers for Disease Control and Prevention