Practical guide
How to report AI capability evidence to an AI transformation leader
An AI transformation leader needs a report that connects observed capability evidence to a bounded programme decision. A useful report does not collapse training completion, tool activity, assessment performance, workplace outcomes, and business impact into one adoption score. It shows what was observed, under which conditions, what evidence is missing, and what action is justified now. Classify.
An AI transformation leader needs a report that connects observed capability evidence to a bounded programme decision. A useful report does not collapse training completion, tool activity, assessment performance, workplace outcomes, and business impact into one adoption score. It shows what was observed, under which conditions, what evidence is missing, and what action is justified now. Classify the transformation leader as the report owner or decision influencer only when that matches the organisation's mandate. A programme sponsor may own funding, functional leaders may own deployment, and security, legal, privacy, or workforce specialists may own separate acceptance decisions.
Start with one reporting decision
Name the audience, reporting period, programme boundary, decision date, and requested action. Examples include repairing a learning module, expanding a controlled pilot, commissioning more evidence, or stopping a workflow. “Show AI progress” is too broad to define evidence or accountability.
NIST's AI Risk Management Framework separates governance, mapping, measurement, and management, with documented roles and ongoing review. The GAO AI Accountability Framework organises evidence around governance, data, performance, and monitoring. These structures support a report that preserves evidence layers and assigns action rather than presenting a decorative dashboard.
Use a SIGNAL report
Build six sections: Scope, Indicators, Gaps, Narrative, Actions, and Limits.
### Scope
State the cohort, eligibility rules, task, environment, time window, assessment version, comparison basis, and excluded uses. Record how many people were invited, started, completed, produced comparable evidence, or were excluded because of technical or protocol failures.
Do not describe a voluntary or convenience sample as the workforce. Keep before-and-after cohorts comparable, and disclose changes to task difficulty, tool access, materials, scoring, or support.
### Indicators
Group evidence by what it can show:
| Layer | Example evidence | Interpretation boundary | | --- | --- | --- | | Participation | invitation, start, completion | Opportunity and participation, not capability | | Activity | tool events, prompts, documents opened | Behaviour occurred, not that it was effective | | Capability | task framing, evidence use, verification, judgment, delivery | Performance in the bounded assessment | | Operating transfer | manager-reviewed workplace artifacts, exceptions, rework | Context-specific use after learning | | Outcome | quality, cycle time, risk, customer or service result | Requires a separate design and credible comparison |
Show definitions, numerator, denominator, source, owner, missingness, and direction for each indicator. Do not average unlike constructs into one number.
### Gaps
List missing records, non-comparable sessions, small groups, scoring uncertainty, inaccessible evidence, and confounds. State whether each gap blocks the decision, narrows it, or becomes a follow-up question. Missing evidence is not poor participant performance.
### Narrative
Explain the material pattern, strongest counterexample, and plausible alternative explanations. Link every claim to a table, retained artifact, or review record. Use precise language such as “in this simulation” or “among 18 comparable sessions.” Avoid claims about future performance or organisation-wide impact when the evidence covers a bounded task.
### Actions
Give each proposed action an owner, decision right, due date, required evidence, stop condition, and next review. Separate immediate operational repair from later programme evaluation.
### Limits
End with what the report cannot establish. Typical limits include selection effects, synthetic work, short observation windows, directional benchmarks, missing workplace outcomes, and inability to attribute change to one programme component.
Worked artifact: report a verification-skills pilot
A synthetic transformation programme runs a 24-person incident-summary assessment after training. Twenty-one sessions are comparable. The reporting question is whether to repair the module and repeat the pilot before adding another function.
The evidence packet contains final deliverables, source traces, event logs, rubric reviews, technical-incident records, and participant feedback. The report records:
The report does not label a 71% source-trace rate as “AI readiness.” Instead, it says the bounded task exposed a repeated contradiction-handling weakness and a separate instrumentation defect. It recommends:
| Action | Owner | Gate | Next evidence | | --- | --- | --- | --- | | Add contradiction practice | Learning owner | Revised exercise approved | Representative task review | | Repair source-trace capture | Product owner | Fault reproduced and fixed | Technical retest | | Re-score affected sessions | Independent reviewer | Stable evidence available | Reconciled review record | | Repeat pilot | Transformation leader | At least 20 comparable sessions | Same-version assessment packet |
Expansion is held until the repeat produces sufficient comparable evidence and operating owners accept the workflow. Employment, promotion, and individual ranking are outside the report's approved use.
- 24 invited, 24 started, 23 submitted, 21 comparable;
- 18 of 21 comparable submissions framed the intended audience and decision;
- 15 of 21 traced all material incident claims to supplied records;
- six submissions missed a contradiction between a timeline and an executive email;
- two sessions lost source-trace events because of a tool fault; and
- reviewer disagreement affected three initial rubric judgments and was reconciled by a second review.
Handle predictable objections
If leaders ask for one headline number, provide a small set of decision-linked indicators and explain why they are not interchangeable. If tool analytics already exist, use them for participation and activity while preserving the separate capability layer. If a small cohort is challenged, report it as a pilot and avoid subgroup conclusions that the sample cannot support.
If the assessment is described as “not real work,” agree with the boundary: it is a controlled opportunity to observe task-bounded performance. Pair it with separately governed operating evidence when the decision concerns transfer to work.
Review before distribution
Confirm that every denominator is visible, exclusions are explained, task versions are separated, claims trace to evidence, adverse cases are retained, and actions have owners. Check privacy, access, retention, and reporting obligations. Remove personal detail that the decision does not require and do not publish small-group comparisons that create avoidable identification or overinterpretation risk.
Two Prune produces structured assessment evidence for human interpretation. It does not prove programme impact, predict workplace performance, operate an ATS, or recommend employment decisions.
Sources
NIST, AI Risk Management Framework Core
- NIST, Descriptions of AI actor tasks
- U.S. Government Accountability Office, Artificial Intelligence: An Accountability Framework
Sources: [1] [2] [3]
Sources
- 1.NIST, AI Risk Management Framework Core · National Institute of Standards and Technology
- 2.NIST, Descriptions of AI actor tasks · National Institute of Standards and Technology
- 3.U.S. Government Accountability Office, Artificial Intelligence: An Accountability Framework · U.S. Government Accountability Office