How an AI Academy Owner Can Build the Case for Applied AI Capability Evidence

Define the decision and the stakeholder role

Start with the decision the organization must make: whether to fund a pilot, expand an existing academy, change curriculum, or add applied assessment. Name the academy owner as owner, influencer, reviewer, or collaborator for each decision rather than assuming the title carries budget authority.

The OECD policy brief on bridging the AI skills gap reports that workplace demand includes both specialized AI expertise and general AI literacy, while understanding of whether training supply meets current and future needs remains limited. Use that as a reason to investigate the organization's own gap, not as proof that a particular programme will work.

Write a decision brief with five entries:

Avoid framing the case as “employees need more AI.” Specify a work outcome such as producing traceable analyses, verifying generated claims, or escalating uncertainty before delivery.

• target population and the work they actually perform;
• costly or risky tasks that require better AI-enabled practice;
• current evidence, including what is missing;
• proposed intervention and alternatives;
• funding, risk, and operational decision owners.

Sources: [1]

Build the evidence-to-investment chain

Use a five-link chain: work need, observable capability, learning experience, assessment evidence, and operating decision. Each link needs an owner and a falsifiable statement. A curriculum topic should map to a behavior that can be observed in representative work, and the proposed evidence should be strong enough to inform the next programme decision without pretending to prove business impact.

The NIST AI Risk Management Framework is voluntary and intended to help organizations incorporate trustworthiness considerations into the design, development, use, and evaluation of AI systems. Its Govern, Map, Measure, and Manage functions can prompt risk and oversight questions for an academy pilot. The NIST AI RMF Playbook also states that its suggested actions are not a universal checklist. Select only the actions relevant to the work context and document the choice.

For each proposed module, record the task, source materials, expected deliverable, risk traps, observable process evidence, scoring rule, and escalation boundary. Compare at least three options: no change, content-only training, and training plus realistic practice and assessment. Include delivery cost, participant time, facilitator time, assessment operations, accessibility work, and follow-up analysis.

Sources: [2] [3]

Worked pilot case

Suppose an academy supports 120 operations managers who increasingly use generative AI to draft incident summaries and action plans. Leaders report inconsistent source checking, but they have no common evidence about how managers work. The academy owner proposes a 24-person pilot rather than asserting an organization-wide deficit.

The pilot contains a short policy refresher, a coached practice case, and a timed simulation using synthetic incident records. Participants must frame the task, locate conflicting evidence, use AI support, verify dates and claims, document uncertainty, and deliver an action note. Reviewers score the same six published capability dimensions under a fixed rubric and capture evidence strength and scoring disagreements.

The pre-agreed decision thresholds are operational, not promotional: at least 20 complete attempts; at least 80 percent reviewer agreement within one rating band; no unresolved access or data-handling incident; and enough process evidence to explain the score for each attempt. The team also records completion time, participant feedback, facilitator effort, accessibility issues, and the proportion of evidence marked insufficient.

If those gates pass, leaders decide whether to run a larger calibration cohort. If reviewer agreement is poor, the next investment is rubric and assessor improvement, not wider rollout. If scores improve after coaching, that is still not proof of productivity gain, risk reduction, or transfer to live work. A separate operating study would be needed for those claims.

Present costs, objections, and limits

Present the case as a decision table with options, total cost, expected evidence, risks, dependencies, and stop conditions. Include the strongest objections: the target task may be wrong, participation may be selective, a simulation may not transfer to live work, reviewers may disagree, and platform telemetry may be incomplete. State how the pilot will detect each failure rather than dismissing it.

Report observed assessment results separately from course satisfaction, tool usage, and business outcomes. Do not turn attendance or prompt counts into capability measures. Preserve missing data as a confidence limitation, and keep benchmark language directional until a relevant comparison group is sufficiently established.

Two Prune can support the simulation and evidence-backed assessment portion of this plan. It does not decide who receives employment opportunities, certify an academy, promise business results, or replace the organization's budget, risk, learning, and operational owners.