AI Capability Pilot Design for an AI Academy Owner

Fix the pilot question and authority map

Write one decision question before selecting content. Examples include whether to revise a module, expand to another calibration cohort, change the assessment rubric, or stop because evidence quality is inadequate. Name the academy owner as owner, influencer, reviewer, or collaborator for curriculum, budget, data access, risk, accessibility, assessment, and operational decisions. Do not assume that the role controls all of them.

The OECD brief on bridging the AI skills gap distinguishes specialized AI expertise from broader AI literacy and notes limited understanding of whether training supply meets current and future needs. The newer OECD Skills in the AI Age report emphasizes labour-market relevance, access, lifelong learning, and shared responsibility. These sources support investigating a local need; they do not establish that a particular academy format will cause a business outcome.

Create an authority map with a single accountable person for each approval and a consultation path for affected teams. Include a “no decision” outcome so weak evidence cannot be converted into an expansion recommendation by default.

Sources: [1] [2]

Build the task-to-evidence protocol

Choose one representative work product rather than testing generic familiarity with tool features. Define the input materials, intended audience, constraints, allowed AI tools, expected deliverable, risk traps, time limit, and observable behaviors. Map each scored behavior to evidence that can actually be captured: source opening, prompt iteration, verification action, revision, calculation, escalation, or final delivery.

Use the voluntary NIST AI Risk Management Framework to prompt governance, context mapping, measurement, and response questions. The NIST AI RMF Playbook explicitly offers suggested actions rather than a universal checklist. Select the actions relevant to the pilot and record why others do not apply.

The protocol should specify:

Keep training participation, tool usage, assessment performance, and operating outcomes as separate measures. A participant can attend a module without demonstrating the behavior, and a strong simulation result does not establish live-work impact.

• the target population and inclusion route;
• the synthetic or approved source materials;
• the intervention sequence and facilitator script;
• assessment instructions and permitted assistance;
• observable evidence for every scoring criterion;
• scorer training, calibration, and disagreement handling;
• missing-data and incomplete-attempt rules;
• privacy, access, accessibility, and incident procedures;
• analysis plan, decision thresholds, stop conditions, and owners.

Sources: [3] [4]

Set sample and comparison rules before launch

A small operational pilot can expose feasibility problems, but its sample should match the decision. Record how people are invited, who opts out, prior experience, relevant work context, and incomplete attempts. If the pilot compares two formats, assign them using a documented rule and keep task difficulty, time, scorer process, and evidence capture equivalent. If assignment is not random, label the comparison descriptive and list plausible confounds.

Define decision gates in advance. Useful gates concern evidence completeness, scorer agreement, accessibility, incidents, delivery effort, and participant comprehension. Avoid selecting thresholds after seeing the results. Specify the denominator for every percentage and preserve missingness rather than silently excluding it.

Plan a scorer-calibration sample before scoring the full cohort. Two reviewers should independently score the same attempts, record disagreements by criterion, and determine whether the rubric or evidence is the problem. A high average score cannot rescue a rubric that reviewers apply inconsistently.

Worked 30-person pilot

An academy owner is asked to improve how service-operations leads use AI when summarizing incidents. The owner recruits 30 volunteers across three regions and records that the sample is self-selected. Each participant receives a short policy module, completes a coached practice, and then enters a timed simulation with synthetic tickets, a service-level report, a conflicting manager email, and an outdated draft note.

Participants must frame the audience and decision, locate the date conflict, use AI to organize evidence, verify generated claims, calculate the affected customer count, record uncertainty, and submit an action brief. The academy captures process evidence and the final artifact. Ten attempts are double-scored before the remainder is assigned.

The pre-registered gates are: at least 24 complete attempts; source evidence available for at least 90 percent of scored criteria; reviewers within one rating band on at least 80 percent of double-scored criteria; no unresolved data-handling or access incident; and facilitator effort below 12 hours for the cohort. Expansion is not automatic. Passing all gates authorizes a larger calibration cohort; missing evidence sends the instrumentation back for repair; poor scorer agreement pauses scoring; an unresolved incident stops the pilot.

Results show 27 completions, 93 percent evidence availability, 76 percent agreement, no incident, and nine facilitator hours. The pilot therefore does not advance. The owner convenes reviewers, finds that “verification” combines arithmetic checking and source contradiction handling, splits the criterion, adds anchor examples, and schedules a second calibration set. The result says nothing yet about productivity, business return, or workplace transfer.

Close with a decision packet

Produce a packet containing the protocol version, authority map, invitation and inclusion record, materials, assessment rubric, scorer calibration results, incident log, accessibility findings, missing-data table, delivery effort, results by pre-specified measure, deviations, and decision record. Report denominators and uncertainty plainly. Preserve participant feedback as experience evidence, not capability evidence.

The academy owner can coordinate the pilot and recommend the next evidence step. Budget owners, risk owners, operational leaders, and affected people retain their own decisions. Two Prune can support the realistic simulation and evidence-backed assessment, but it does not certify an academy, promise operating results, or make employment decisions.