Practical guide

AI capability pilot design for a Head of AI

A Head of AI often needs to learn whether people can use AI well in real work—not merely whether they attended training or liked a tool. A useful pilot therefore tests a bounded workplace task, preserves evidence of the work process, and gives leaders a pre-agreed decision at the end. This guide uses the PILOT method to design that experiment without treating the Head of AI as the universal.

By Two Prune

A Head of AI often needs to learn whether people can use AI well in real work—not merely whether they attended training or liked a tool. A useful pilot therefore tests a bounded workplace task, preserves evidence of the work process, and gives leaders a pre-agreed decision at the end. This guide uses the PILOT method to design that experiment without treating the Head of AI as the universal owner of budget, security, talent decisions, or business outcomes.

Clarify the Head of AI mandate

The title varies widely. Before designing anything, classify the Head of AI’s role for this decision:

| Decision area | Typical pilot role | Accountable counterpart | |---|---|---| | AI capability question and portfolio fit | Owner or influencer | Executive sponsor | | Quality of the simulated work task | Collaborator | Business-function owner | | Security, privacy, and tool access | Collaborator | Security and data owners | | Assessment interpretation | Reviewer | Assessment owner or people lead | | Pilot spend and scale decision | Influencer or reviewer | Named budget owner |

This classification must be confirmed locally. NIST’s descriptions of AI actor tasks are useful because they separate responsibilities across executive leadership, governance, operations, domain experts, users, and evaluators. The design should assign work to real owners rather than collapse those responsibilities into one AI title.

Also record the current alternative. It may be a self-report survey, course completion, a generic AI quiz, manager observation, or no structured evidence. The pilot should be judged against that alternative, not against an imaginary perfect system.

Sources: [1]

Design the pilot with PILOT

### P — Purpose and terminal decision

Write one question the pilot can answer and one decision it cannot. For example:

> Can a realistic assessment provide interpretable evidence about how a defined operations cohort frames, researches, verifies, and communicates an AI-enabled work task?

The terminal decision might be to stop, revise, or run a larger validation cohort. It should not be “prove the workforce is AI-ready.” A small pilot cannot establish a mature organization-wide benchmark or a causal productivity effect.

Define the comparison before launch. If the current alternative is a confidence survey, compare evidence completeness, reviewer usefulness, participant burden, delivery effort, and unresolved risk—not just average scores.

### I — Inputs and inclusion rules

Freeze the workflow, cohort, materials, and intended output. Choose a real task pattern but use synthetic or properly authorized materials. Record:

The OECD employer and worker surveys on AI illustrate why local evidence matters: exposure, use, training, and perceived effects differ by workplace and worker context. Do not turn a broad population finding into a presumed result for this cohort.

### L — Live assessment and evidence map

Map every decision to observable evidence before anyone participates.

| Pilot question | Evidence | Review rule | Failure condition | |---|---|---|---| | Can participants frame the task? | Brief interpretation, constraints, success criteria | Rubric anchored to the specific case | Rubric cannot distinguish task reading from generic prose | | Can they navigate evidence? | Opened sources, selected facts, contradictions addressed | Trace claims to case materials | Source events are missing or ambiguous | | Can they orchestrate AI? | Prompts, iterations, critiques, revisions | Examine purposeful use, not prompt count | Tool telemetry is incomplete or changes by participant | | Can they control risk? | Checks, corrected errors, uncertainty labels | Review both process and deliverable | Reviewers cannot reproduce the evidence link | | Is the output useful? | Final business deliverable | Business owner applies case-specific standard | Task does not resemble a meaningful work decision |

Two Prune assessments can capture evidence across six pillars: context framing, evidence navigation, AI orchestration, verification and risk control, judgment and synthesis, and communication and delivery. The pilot should test whether those observations are useful for the stated decision; it should not assume a score alone answers it.

### O — Oversight, objections, and stop gates

Create a small review group: Head of AI, business-function owner, assessment owner, and the relevant security or data reviewer. Add a participant-support contact who is not also scoring the work.

Resolve common objections explicitly:

Stop or pause if a security review is incomplete, participants receive materially different conditions, event evidence is missing, the rubric changes after results are seen, or reviewers cannot explain a score from retained evidence. NIST’s AI RMF Core supports this lifecycle discipline through Govern, Map, Measure, and Manage functions rather than a one-time approval.

### T — Thresholds and terminal packet

Pre-register thresholds as decision rules, not marketing targets. A pilot might require:

Thresholds are local operating criteria. They are not proof of scientific validity, legal suitability, or future business performance.

  • the participating function and why the task matters;
  • inclusion and exclusion rules for the cohort;
  • the expected deliverable and time limit;
  • tools and AI access available to every participant;
  • required source materials and planted evidence traps;
  • consent, access, retention, and support arrangements;
  • what prior training or experience will be captured as context.
  • “The case is artificial.” Ask the function owner to verify the task’s decisions, tradeoffs, and deliverable, while keeping confidential production data out of scope.
  • “Scores will be used beyond the pilot.” State the permitted decision, access list, retention period, and deletion or archival route before launch.
  • “Different AI familiarity makes comparison unfair.” Standardize available tools and instructions; record relevant experience; interpret pilot results as evidence from this cohort, not a universal rank.
  • “The assessment will slow delivery.” Set a burden ceiling and measure preparation, participant, reviewer, and support time.
  • “The pilot will prove business value.” Separate assessment usefulness from downstream productivity or commercial outcomes, which require different evidence.
  • at least 90% of completed records to contain the evidence needed for every scored pillar;
  • two reviewers to agree within one rubric level on a blinded sample, with disagreements reconciled and documented;
  • no unresolved critical access, privacy, or scoring incident;
  • median participant burden within the agreed time box;
  • business reviewers to identify a specific decision the report improves over the current alternative.

Sources: [2] [3]

Worked example: a 12-person operations cohort

A Head of AI proposes a two-week pilot for 12 operations managers. The task is to review a synthetic service backlog, conflicting customer messages, and a policy excerpt, then submit a recovery recommendation with calculations and sources. Everyone receives the same approved AI tool and 75-minute window.

The business-function owner approves the case. Security approves the synthetic materials and access. The assessment owner freezes the rubric. The Head of AI coordinates delivery and owns the final portfolio recommendation; the operations director owns any decision to change operational training.

At the terminal review, 11 people complete the task. Ten records have complete evidence; one loses part of the AI interaction log. Reviewer agreement meets the local rule after two anchor descriptions are clarified. Business reviewers find the source-to-claim view more useful than the prior confidence survey, but preparation time exceeds the ceiling by 30%.

The decision is revise, not scale: simplify case setup, rerun with a small cross-functional sample, and exclude the incomplete record from score aggregation while retaining the incident in the pilot log. No claim is made about workforce-wide readiness or productivity.

Deliver the decision packet

The Head of AI’s terminal packet should contain the original purpose, role map, protocol version, cohort and completion counts, evidence completeness, reviewer calibration result, burden and delivery cost, incidents, participant feedback with response count, comparison with the current alternative, threshold disposition, and a stop/revise/expand recommendation. Keep aggregate observations separate from identifiable participant records.

Two Prune is an assessment product that supports realistic AI-enabled work simulations and evidence-backed performance reports. It is not an ATS and does not make automated employment decisions. Customers remain responsible for lawful use, access, interpretation, and any downstream talent, training, or operating decision.

Sources

  1. 1.descriptions of AI actor tasks · National Institute of Standards and Technology
  2. 2.OECD employer and worker surveys on AI · www.oecd.org
  3. 3.AI RMF Core · National Institute of Standards and Technology