Practical guide

How to report AI capability evidence to a Head of AI

A Head of AI needs a reporting packet that supports an operating decision, not a leaderboard or a collection of training statistics. The packet should connect a defined cohort and work task to observable evidence, evidence quality, ownership, exceptions, and a next decision. It must also separate assessment findings from AI adoption, productivity, business impact, and employment judgments, which.

By Two Prune

A Head of AI needs a reporting packet that supports an operating decision, not a leaderboard or a collection of training statistics. The packet should connect a defined cohort and work task to observable evidence, evidence quality, ownership, exceptions, and a next decision. It must also separate assessment findings from AI adoption, productivity, business impact, and employment judgments, which require different evidence.

Confirm the reporting mandate

Classify the Head of AI’s role for each decision before choosing metrics.

| Decision | Head of AI role | Other accountable owner | |---|---|---| | portfolio evidence standard | owner or influencer | executive sponsor | | task quality and relevance | reviewer or collaborator | business-function owner | | assessment method and scoring | reviewer | assessment owner | | data access and retention | collaborator | security, privacy, and data owners | | training or workflow change | influencer | function or learning owner | | budget or rollout | influencer or reviewer | named budget owner |

NIST’s descriptions of AI actor tasks separate responsibilities among executive leadership, governance, domain experts, operations, users, and test and evaluation roles. Use that separation to prevent the report from assigning every unresolved action to the AI leader.

Also name the current alternative: course completion, tool activity, confidence surveys, manager anecdotes, assessment evidence, or no recurring report. The new packet should explain which decision it improves over that alternative.

Sources: [1]

Build the packet with FRAME

### F — Frame one decision and reporting period

Open with the decision, scope, cohort, assessment version, reporting period, and owner. Examples include:

Do not open with an average score. A number has no operating meaning until the reader knows what was measured, under which conditions, and what decision it informs.

### R — Reconcile the population and denominator

Show invited, started, completed, excluded, and analyzed records. For each exclusion, give a reason such as interrupted session, accessibility failure, incomplete event evidence, wrong assessment version, or withdrawal. Missing evidence must not be scored as poor performance.

Use a cohort table:

| Population stage | Count | Evidence status | Treatment | |---|---:|---|---| | invited | | roster reconciled | context only | | started | | session record present | completion denominator stated | | completed | | submission present | eligibility review | | excluded | | reason recorded | not aggregated | | analyzed | | required evidence present | included in findings |

The OECD’s employer and worker surveys on AI gather complementary organizational and worker perspectives. That is a useful reminder that activity, employer policy, self-reported experience, and observed task performance answer different questions. Do not merge them into one “AI readiness” score.

### A — Anchor findings to evidence

For every headline finding, include the assessment task, rubric or measure version, observed evidence, aggregation rule, missingness rule, and confidence limitation. A compact decision map is more useful than a dense dashboard:

| Decision question | Evidence | Finding | Strength | Limitation | Owner | |---|---|---|---|---|---| | Can participants verify material claims? | source checks, corrections, final citations | | strong / mixed / weak | | assessment owner | | Is the task representative? | domain review and participant debrief | | | | function owner | | Can reviewers apply the rubric? | blinded agreement and adjudication | | | | assessment owner | | Is delivery operable? | completion, support, review time, incidents | | | | programme owner |

Report the six Two Prune pillars only when the assessment actually captured the required evidence: context framing, evidence navigation, AI orchestration, verification and risk control, judgment and synthesis, and communication and delivery. Preserve contrary examples and null findings; do not reduce a severe evidence failure to a reassuring average.

### M — Map action to the right owner

Translate each finding into one bounded action, accountable owner, evidence of completion, and review date. Distinguish capability interventions from other repairs:

The NIST AI RMF Core connects governance, context mapping, measurement, and risk treatment. Its functions are not a reporting template, but they reinforce the need to connect a measure to context and an owned management response.

### E — Expose exceptions and the decision boundary

End with unresolved exceptions, incidents, evidence gaps, changes since the prior period, and the explicit next decision. State what the report cannot establish. Assessment evidence from a bounded simulation does not by itself establish productivity, general job performance, organizational adoption, causal training impact, or financial return.

GAO’s evidence-based policymaking practices emphasize acquiring, assessing, and applying evidence to decisions. Apply that discipline by retaining the decision trail: what evidence was available, how it was judged, what action followed, and what new evidence is expected.

  • repair the assessment task before another cohort;
  • target practice at a specific evidence-navigation failure;
  • investigate an approved-tool access problem;
  • run a broader validation cohort;
  • stop because the evidence is incomplete or the intended use has changed.
  • repository or source-quality failures need knowledge-management work;
  • tool-access failures need IT or security action;
  • unclear approval authority needs workflow governance;
  • invalid case materials need assessment repair;
  • weak observed practice may support targeted practice or coaching;
  • uncertain scoring may require reviewer calibration, not participant remediation.

Sources: [2] [3] [4]

Worked example: a quarterly assessment evidence packet

A synthetic operations cohort has 30 invitations, 27 starts, and 24 submissions. One submitted session lost part of its event record during an outage, so 23 records enter the aggregate. The report does not describe the missing record as a low score.

Observed findings:

The packet recommends repair, not broad rollout. The assessment owner clarifies two verification anchors and reruns a blinded calibration sample. The function owner improves the conflicting-date cue because it was less realistic than intended. The programme owner tests whether the evidence view can reduce review time. The Head of AI owns the portfolio decision record and will review these three artifacts before deciding on another cohort.

The summary reads:

text Decision: repair and retest before expansion Population: 23 analyzed of 24 submitted; one excluded for missing process evidence Strongest signal: task framing under the defined case Material limitation: mixed reviewer agreement on verification behaviors Operating issue: review time 18% above the local ceiling Next gate: revised anchors, calibration result, and review-time retest Not established: workforce readiness, productivity, or financial return

  • 19 of 23 records identify the correct decision and audience;
  • 14 reconcile conflicting source dates before drafting;
  • 11 independently check the calculation that drives the recommendation;
  • reviewer agreement is strong on four pillars but mixed on verification and risk control;
  • median review time exceeds the operating ceiling by 18%;
  • the self-reported confidence survey has 16 responses and is shown separately.

Reporting failure conditions

Hold the packet if cohort denominators do not reconcile, assessment versions are mixed without treatment, missing telemetry is scored as failure, reviewers cannot trace findings to evidence, individual and aggregate access are not governed, or stakeholders intend to use the report outside its approved purpose. Report adverse results rather than changing thresholds after seeing them.

Two Prune is an assessment product. It can provide evidence-backed reports from realistic AI-enabled work simulations for human interpretation. It is not an ATS, does not issue employment recommendations, and does not establish that a participant or organization will achieve future business outcomes.

Sources

  1. 1.descriptions of AI actor tasks · National Institute of Standards and Technology
  2. 2.employer and worker surveys on AI · www.oecd.org
  3. 3.NIST AI RMF Core · National Institute of Standards and Technology
  4. 4.evidence-based policymaking practices · U.S. Government Accountability Office