AI Capability Pilot Design for an AI Enablement Leader

Start with the decision the pilot must unlock

Write a decision card containing the decision due, accountable owner, enablement leader’s role, population, workflow, current alternative, permitted tools and data, evidence required, decision date, and actions explicitly outside scope. Classify the leader as owner, influencer, reviewer, or collaborator rather than inferring authority from title.

Good pilot questions are narrow: can reviewers consistently observe source reconciliation in a defined task; does a practice-and-feedback intervention produce a repeatable task-bounded change; or which barrier prevents a team from using an approved workflow safely? “Does AI training work?” is too broad to design or answer responsibly.

The NIST AI RMF Core emphasizes documented context, roles, human oversight, measurement, monitoring, and risk treatment. It is voluntary guidance and does not prescribe one pilot or establish an organization’s decision rights.

Sources: [1]

Define the workflow and counterfactual

Map the current workflow before choosing the intervention. Record inputs, expected artifact, required judgment, existing tools, controls, handoffs, failure costs, and non-AI alternatives. Separate access, knowledge, applied capability, process design, and governance barriers. A pilot cannot show a capability benefit if participants lack an approved environment or if review standards conflict.

Choose the comparison that matches the decision. Options include a pre-specified baseline task, a later repeat task of comparable difficulty, a matched comparison group, a phased rollout, or a non-training process alternative. Explain allocation, likely selection effects, contamination, and what inference remains unavailable. Do not improvise a comparison after seeing results.

The OECD’s Bridging the AI skills gap discusses general, advanced, and complementary skills along with accessibility and training-system considerations. Use it to frame skill types and participation barriers, not as evidence that a local intervention will succeed.

Sources: [2]

Build the recruitment and evidence funnel

Define eligibility, invitation, consent or notice route where applicable, accommodations, scheduling, tool access, and withdrawal handling. Preserve the funnel from eligible to invited, started, submitted, technically valid, and reviewable. Report reasons for exclusion and missingness without turning nonparticipation into poor performance.

Specify what the pilot captures: final artifact, allowed workspace events, source use, AI interaction, revisions, timing, reviewer notes, and participant feedback. Minimize collection to the decision need. Set access, retention, correction, incident, and deletion routes before launch. Raw work should remain access-controlled.

Set a minimum viable evidence rule rather than a target pass rate. Examples include representative task completion, sufficient reviewable records, reliable event capture, acceptable assessor agreement on predeclared criteria, and no unresolved high-severity data issue. If the rule is missed, the pilot may still yield design learning but cannot support the intended decision.

Calibrate the task and reviewers

Write the task contract: audience, constraints, success criteria, source packet, allowed AI use, expected deliverable, time limit, intentional ambiguity, evidence traps, and failure conditions. Pilot the pilot with a few synthetic or authorized records to catch broken links, impossible instructions, logging gaps, inaccessible materials, and scoring criteria that reward style over evidence.

Create behavioral anchors for each criterion and run reviewer calibration on shared samples. Record disagreements, adjudication, criterion revisions, and version changes. Freeze the scoring version before the main cohort. A high average score is not useful if reviewers apply the rubric differently or the task fails to elicit the behavior of interest.

The OECD’s 2026 brief AI and skills: What we know so far describes changing workplace skill needs and evidence limitations. It supports continued measurement questions, not a local benchmark or causal claim.

Sources: [3]

Pre-commit to decision rules and stop conditions

Write the analysis and decision table before launch. Include metric or evidence item, denominator, subgroup or accessibility check, missing-data rule, threshold or qualitative gate, owner, possible disposition, and next evidence need. Keep access, activation, task performance, reviewer agreement, artifact quality, risk events, participant experience, and workplace outcomes in separate rows.

Use dispositions such as continue to a repeat test, repair the task, redirect to a process change, stop, or not answerable. Stop conditions may include unauthorized data exposure, unreliable event capture, inaccessible participation route, severe distress, unresolvable reviewer disagreement, or evidence cost above the approved boundary. Specify who can pause the pilot and how affected records will be handled.

Do not treat attendance, satisfaction, or tool use as applied capability. Do not treat a task-bounded score change as workplace transfer, productivity, or organization-wide readiness. A later transfer study needs its own design and authority.

Worked pilot: verification practice for customer operations

A synthetic company has 240 customer-operations staff. Discovery identifies three barriers: 60 people lack an approved AI environment, team leads use conflicting review standards, and sampled artifacts often repeat unsupported policy claims. The enablement leader is owner only for the verification-practice pilot; technology owns access and operations owns the review standard.

The decision question is whether a synthetic support task can expose source-checking behavior consistently enough to justify a repeat practice wave. Seventy-two eligible volunteers are invited. Fifty-eight start, 52 submit, and four records are incomplete after a logging outage, leaving 48 reviewable attempts. Those funnel numbers remain visible.

Two reviewers score source selection, contradiction handling, claim verification, escalation, and final communication. Disagreement clusters in the escalation criterion and exceeds the predeclared calibration boundary. Thirty-one of 48 reviewable attempts repeat at least one unsupported claim; 19 of those opened the conflicting source but did not reconcile it. These are task observations, not a workforce prevalence estimate.

The disposition is repair and repeat, not expand. The leader revises the escalation anchor, adds contradiction practice, fixes logging, and keeps access and review-standard work with their actual owners. Company-wide rollout remains out of scope because the volunteer sample, synthetic task, and absent transfer design cannot support it. The retained artifact contains the decision card, workflow map, comparison rationale, recruitment funnel, task and rubric versions, calibration record, incident log, distributions, limitations, and signed next-gate decision.

Keep Two Prune within its boundary

Two Prune can provide a realistic assessment session, submitted work, captured process signals, and evidence-backed scoring for the defined task. It does not prove programme effect, workplace transfer, productivity, general capability, organizational readiness, regulatory compliance, or suitability for employment. Accountable owners must interpret assessment evidence alongside access, workplace, learning, and governance evidence.