Practical guide

How to report AI capability evidence to a Chief Learning Officer

Report AI capability evidence to a Chief Learning Officer as a decision packet, not a score dashboard. The packet should show who participated, what work was observed, what the evidence supports, where it is incomplete, what operating burden occurred, and which learning or measurement decision follows. It must keep task performance separate from course completion, workplace transfer, and.

By Two Prune

Report AI capability evidence to a Chief Learning Officer as a decision packet, not a score dashboard. The packet should show who participated, what work was observed, what the evidence supports, where it is incomplete, what operating burden occurred, and which learning or measurement decision follows. It must keep task performance separate from course completion, workplace transfer, and business outcomes.

Name the decision and the CLO’s role

Classify the Chief Learning Officer as owner, influencer, reviewer, or collaborator for the specific decision. A CLO may own learning design while privacy, security, business operations, finance, or another executive owns related controls and investment. Do not infer budget or employment authority from the title.

Before building the report, record its primary user, decision date, alternatives, population, evidence window, and prohibited interpretations. Examples include revise a learning pathway, add practice, commission a transfer study, investigate a workflow barrier, or stop collecting a measure that does not inform action.

OPM’s training evaluation guidance distinguishes planning, implementation, and evaluation and notes that training is not always the answer to a performance gap. CDC’s evaluation guidance recommends identifying users, questions, measures, collection methods, analysis, communication, and use in advance. Its Quality Training Standards separately address needs, objectives, learner assessment, evaluation, and follow-up support. These sources support disciplined reporting but do not prove that a programme or assessment caused an operational outcome. OPM: Planning and evaluating training CDC: Building a training evaluation plan CDC: Quality Training Standards

Sources: [1] [2] [3]

Use the SIGNAL reporting method

### S — Specify the decision, users, and evidence boundary

Lead with the decision requested and the report’s valid scope. State the assessment or learning context, dates, population, participation rule, task conditions, and evidence cutoff. Name secondary users and access restrictions.

Include a one-sentence boundary: “This report describes observed performance in the stated task; it does not by itself establish workplace transfer, business impact, or suitability for another purpose.”

### I — Inventory participation and evidence flow

Show invitations, eligible participants, access, starts, completions, withdrawals, technical failures, accommodations, excluded records, and missing evidence. Do not collapse these states into a completion percentage.

Map each reported measure to its source, owner, collection time, inclusion rule, calculation, quality check, and retention route. Mark self-report, knowledge checks, observed task evidence, manager observation, work samples, and operational data as different evidence types.

### G — Group results without hiding variation

Report distributions, not only averages. Where appropriate and permitted, show pre-specified groups, task versions, evidence completeness, reviewer agreement, and relevant failure patterns. Suppress or combine small groups when reporting could expose individuals or create unstable comparisons.

Use a failure taxonomy that helps a learning decision: framing, source selection, AI prompting, verification, calculation, risk handling, synthesis, and communication. Avoid a single composite score when different remedies are needed.

### N — Narrate findings, alternatives, and uncertainty

For each material finding, use four lines:

Distinguish evidence of reach, participation, learning, simulated task performance, transfer, and operational outcomes. A task result can inform capability development without showing that behaviour appears later at work.

### A — Assign actions, owners, and thresholds

Translate findings into options: maintain, revise content, add practice, change support, repair the task or rubric, investigate a non-learning cause, test transfer, expand the sample, hold, or stop. Give each action an owner, due date, evidence requirement, and reconsideration threshold.

Do not set a favourable threshold after seeing results. If a previously defined threshold was wrong, preserve it and document the change for the next cycle.

### L — Log limitations, objections, and next review

List sampling limits, missing data, task fidelity, measurement error, reviewer disagreement, technical incidents, accessibility issues, and attribution limits. Record the strongest objections and their dispositions. Name the next report date and the evidence that could change the conclusion.

  • Observed: the exact pattern and population.
  • Supported interpretation: what the design permits.
  • Alternative explanation: another plausible cause.
  • Decision relevance: which option it informs.

Executive evidence card

| Field | Required content | | --- | --- | | Decision | exact choice, owner, and date | | Population | eligible, invited, started, completed, missing, excluded | | Observed work | task, tools, evidence, constraints, deliverable | | Findings | distributions, failure patterns, evidence strength | | Interpretation | supported meaning and alternative explanations | | Operations | setup, reviewer time, support, incidents, exceptions | | Safeguards | purpose, access, retention, accessibility, human review | | Options | maintain, revise, investigate, test, hold, stop | | Recommendation | rationale, owner, threshold, next evidence gate |

Attach the measure dictionary, task and rubric versions, calculation file, reviewer agreement and adjudication record, exception log, and approved audience list. The card is a navigation layer, not a substitute for those artifacts.

Worked report: verification practice after an AI course

A synthetic learning team reports on 42 analysts who were eligible for a verification module. Thirty-eight started, 34 completed the controlled task, two encountered technical failures, one used an accommodation, and four did not start. The report preserves these states rather than saying “81% completed.”

The task asks participants to prepare a recommendation using a spreadsheet, two dated source notes, and an AI assistant. It contains an unsupported citation, a denominator mismatch, and a later source that contradicts an earlier figure. Two reviewers score a pre-registered sample independently.

The executive card reports that most completers flag the citation, fewer recalculate the denominator, and a smaller group explains how the later source changes the recommendation. It does not compare unnamed teams because the groups are small and participation differs. Reviewer disagreement is concentrated in one rubric item that combines calculation accuracy with communication.

The supported interpretation is that the task exposed uneven verification performance under controlled conditions. Alternative explanations include unfamiliar spreadsheet structure, self-selection, and the ambiguous rubric item. Course completion, satisfaction, and task performance are shown in separate rows.

The recommendation is revise and retest: split the rubric item, add denominator practice, repair the spreadsheet instructions, and repeat with a pre-specified sample. The CLO is the learning-design owner; the analytics lead owns the calculation check, accessibility reviews the revised task, and business managers own any later transfer sampling. No individual result is presented as an employment decision.

The evidence packet records population flow, versions, source set, scores, disagreement, adjudication, technical failures, actions, owners, and the next review date. It explicitly does not claim workplace transfer, productivity improvement, return on investment, or organization-wide capability.

Failure and hold conditions

Hold the report when the decision is unstated, participation states are collapsed, a score cannot be reproduced, group comparisons were invented after collection, small cells expose people, missing evidence is treated as poor performance, task evidence is called workplace transfer, or the recommended action has no owner and next test.

Two Prune provides realistic AI-enabled work assessments and evidence about observed task performance. It can support a customer’s learning evaluation, but it does not deliver training, prove causal impact, or make employment decisions.

Sources

U.S. Office of Personnel Management: Planning and evaluating training

  • Centers for Disease Control and Prevention: Building a training evaluation plan
  • Centers for Disease Control and Prevention: Quality Training Standards

Sources: [1] [2] [3]

Sources

  1. 1.OPM: Planning and evaluating training · U.S. Office of Personnel Management
  2. 2.CDC: Building a training evaluation plan · Centers for Disease Control and Prevention
  3. 3.CDC: Quality Training Standards · Centers for Disease Control and Prevention