Practical guide
AI capability pilot design for an AI transformation leader
An AI transformation leader needs a pilot that can answer a bounded capability decision, not a showcase that rewards novelty. The pilot should reveal whether people can perform representative work with AI, verify the result, manage risk, and communicate a useful deliverable under stated conditions. It should also show where the operating system around the work is incomplete. Classify the.
An AI transformation leader needs a pilot that can answer a bounded capability decision, not a showcase that rewards novelty. The pilot should reveal whether people can perform representative work with AI, verify the result, manage risk, and communicate a useful deliverable under stated conditions. It should also show where the operating system around the work is incomplete. Classify the transformation leader as the pilot owner when they control the learning question, design, evidence review, and recommendation. Budget approval, employment decisions, information-security acceptance, or operational rollout may belong to other authorities. Name those boundaries before recruiting participants.
Use the PILOT design record
Build one record with five linked parts: Purpose, Inclusion, Learning task, Observation, and Thresholds.
### Purpose
Write one decision the pilot will inform. For example: “Should the operations function expand structured AI-assisted incident summarisation from a controlled cohort to two additional teams?” List alternatives such as retaining the current process, repairing the workflow, changing the tool or task, repeating the pilot, or stopping. A general goal such as “demonstrate AI value” cannot define success or failure.
NIST's AI Risk Management Framework separates governance, mapping, measurement, and management. Its actor-task descriptions also distinguish responsibilities across people who design, deploy, operate, evaluate, and oversee AI. The GAO AI Accountability Framework similarly organises evidence around governance, data, performance, and monitoring. These sources support a pilot that assigns roles and gathers evidence across the full work system rather than attributing all results to a model.
### Inclusion
Define the eligible work population, sampling rationale, cohort size, relevant experience, accommodations, and exclusions. Include the range of normal cases and known difficult cases. Do not choose only enthusiasts or the easiest documents. If participation affects employment, compensation, or access, obtain the required organisational review and keep pilot evidence separate from automated employment decisions.
Create a responsibility map:
| Role | Pilot responsibility | | --- | --- | | Transformation leader | Owns the learning question, protocol, evidence review, and recommendation | | Process owner | Confirms representative tasks and operational constraints | | Security or privacy reviewer | Reviews approved inputs, tool boundaries, retention, and exceptions | | Assessment designer | Creates comparable simulations and scoring evidence | | Participant | Completes the task and reports material friction or uncertainty | | Independent reviewer | Checks scoring consistency, missing evidence, and adverse cases | | Decision authority | Chooses expand, repair, repeat, or stop |
### Learning task
Choose a realistic task with an observable deliverable and an evidence trail. Freeze the briefing, source materials, tool permissions, time limit, expected deliverable, and evaluation criteria. Include ambiguity and evidence traps that require judgment rather than prompt recall. Separate the capability question from tool usability, access failure, and process defects so the pilot does not blame a participant for a broken environment.
Record current alternatives. These may include manager observation, training completion, self-report, generic AI tests, portfolio review, or a realistic work simulation. State what each alternative can and cannot reveal. A pilot can combine evidence sources; it should not claim one assessment proves future workplace outcomes.
### Observation
Predefine the evidence map before the first session:
Keep final-output scores separate from process observations. Record which evidence supports each finding, which signals are absent, and whether technical failures made a session non-comparable. Two Prune assessments can contribute structured evidence across these dimensions, but the customer remains responsible for the pilot decision and surrounding controls.
### Thresholds
Set thresholds that lead to an action. Include minimum comparable sessions, evidence-completeness rules, material-risk triggers, review consistency, operational feasibility, and stop conditions. Define what happens when results are mixed.
Use four dispositions:
- task interpretation and success criteria;
- documents and data reviewed;
- AI prompts, decomposition, critique, and revision;
- source checks, calculation checks, and risk controls;
- reasoning behind tradeoffs;
- final deliverable quality;
- timing, missingness, tool errors, and requests for support; and
- participant feedback about unclear or unrealistic conditions.
- Expand when the evidence supports the scoped use and the operating controls are ready.
- Repair when a specific task, environment, rubric, training, or control defect can be corrected.
- Repeat when missingness or a protocol change prevents a sound comparison.
- Stop when the task is unsuitable, required controls cannot be met, or material risk exceeds the authority's boundary.
Worked artifact: a 24-person incident-summary pilot
A synthetic transformation team wants to test AI-assisted incident summarisation. The owner selects 24 operations staff across three experience bands. Each completes one of four balanced simulations using the same tool boundary and time limit. The task requires an executive summary, source-linked chronology, unresolved contradictions, and recommended follow-up. AI may draft and compare evidence; it may not invent events, alter incident records, or decide disciplinary action.
The pilot record sets these preconditions:
| Decision element | Rule | | --- | --- | | Comparable evidence | At least 20 sessions with complete event, source, and submission records | | Verification | Material claims must trace to the supplied incident record | | Risk trigger | Any fabricated critical event receives independent review and blocks automatic expansion | | Review consistency | Disputed rubric judgments are reconciled by a second reviewer | | Operational readiness | Approved access, retention, fallback, and support owners must be present | | Reporting | Results are split by task version and experience band; small groups are not overinterpreted |
Twenty-one sessions are comparable. Most deliverables are clear, but six participants fail to resolve a contradiction between the timeline and an executive email. Two tool interruptions also cause missing source traces. The outcome is repair, not expansion: revise the briefing to clarify contradiction handling, correct the source-trace failure, and repeat the affected cases. The record does not translate pilot scores into retention, promotion, or selection decisions.
Report the result without overstating it
Publish the protocol version, cohort definition, comparable-session count, task conditions, evidence completeness, observed failure modes, disposition, accountable authority, and next review. Report uncertainty and missing evidence alongside averages. Avoid claims that the pilot establishes organisation-wide productivity, safety, return on investment, or predictive validity unless those questions were separately designed and supported.
Two Prune is an assessment product for realistic AI-enabled work simulations and evidence-backed scores. It does not operate a hiring pipeline, recommend employment decisions, or prove that a pilot outcome will generalise beyond the tested scope.
Sources
NIST, AI Risk Management Framework Core
- NIST, Descriptions of AI actor tasks
- U.S. Government Accountability Office, Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities
Sources: [1] [2] [3]
Sources
- 1.NIST, AI Risk Management Framework Core · National Institute of Standards and Technology
- 2.NIST, Descriptions of AI actor tasks · National Institute of Standards and Technology
- 3.U.S. Government Accountability Office, Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities · U.S. Government Accountability Office