Practical guide
How a chief human resources officer can design an AI capability pilot
A chief human resources officer can use a bounded assessment pilot to learn whether realistic work evidence supports a capability decision. The PILOT method is to Pick one decision and workflow, Invite a fair and supportable population, Lock the evidence protocol, Operate the simulation with safeguards, and Test decision value against pre-agreed continue, change, or stop rules.
A chief human resources officer can use a bounded assessment pilot to learn whether realistic work evidence supports a capability decision. The PILOT method is to Pick one decision and workflow, Invite a fair and supportable population, Lock the evidence protocol, Operate the simulation with safeguards, and Test decision value against pre-agreed continue, change, or stop rules.
P — Pick the decision and a representative workflow
Write the pilot decision in one sentence. A useful example is: “Should we adopt this assessment design to identify team-level verification needs for employees who use AI to summarize customer policy changes?” Avoid questions such as “Does AI work?” or “Who is good at AI?” They do not define a workflow, evidence gap, or action.
Choose a workflow with:
Map alternatives before selecting a pilot. Training completion can show participation, a self-report can reveal confidence, tool telemetry can show activity, and manager observation can provide context. None alone reconstructs performance on a realistic task. An assessment simulation is useful when the missing evidence concerns how work is performed, but it should not be presented as a replacement for all other evidence.
CIPD’s current guide recommends connecting AI skills planning to workflows, involving employees, starting with controlled pilots and clear measures, and using shorter review cycles as technology changes. This supports a staged design; it does not prove that a particular assessment improves business performance. CIPD guide to AI skills planning
- a real audience and a recognizable business deliverable;
- enough source material to require evidence navigation and judgment;
- a safe synthetic or anonymized simulation environment;
- observable opportunities to frame, prompt, verify, revise, and communicate;
- no dependency on live customer, employment, or regulated decisions;
- a business owner who can judge whether the simulation is representative.
Sources: [1]
I — Invite a fair and supportable population
Define eligibility before invitations are sent. Record the business unit, workflow exposure, role context, tenure or access requirements that genuinely affect the task, and explicit exclusions. Do not select only enthusiastic tool users and then generalize to the whole workforce.
The invitation record should include:
| Field | Pilot requirement | | --- | --- | | eligible population | rule and count established before recruitment | | selection method | census, random sample, stratified sample, or justified purposive sample | | participation | voluntary/required status, purpose, time, support, and consequences | | accessibility | accommodation route, technology check, and alternative timing | | privacy | collected data, permitted use, access, retention, and deletion route | | incomplete attempts | predefined treatment of withdrawals, outages, and access failures | | representation | comparison of invited, started, completed, and excluded groups |
The U.S. Department of Labor’s AI principles and best practices emphasize worker input, human oversight, training, rights, and protection of worker data. They are a useful government reference for participation and governance questions, while applicable law and policy still require qualified review in the organisation’s jurisdiction. U.S. Department of Labor AI best practices
Keep assessment results out of individual employment decisions unless a separate, properly governed use has been approved and validated. A capability pilot can support team-level learning without turning Two Prune into an applicant or employment decision system.
Sources: [2]
L — Lock the evidence protocol before launch
Pre-register what the pilot will measure and how each result could change a decision. Separate four evidence layers:
Define rubric versions, scorer preparation, quality checks, minimum evidence for a score, treatment of missing process signals, and a disagreement route. Freeze the scenario and rubric for the evaluated cohort. If a material defect requires a change, version it and do not compare unlike attempts without disclosure.
Predefine pilot gates rather than inventing a success story after results arrive.
| Gate | Continue | Change | Stop or hold | | --- | --- | --- | --- | | access | completion conditions work across intended groups | repair a bounded access issue and retest | systemic exclusion cannot be resolved | | evidence | reports cite reconstructable process and deliverable evidence | one criterion or artifact needs repair | scores rely on unavailable or invented evidence | | scoring | reviewers apply criteria consistently enough for the decision | retrain, clarify, and double-score | unresolved material disagreement | | actionability | findings map to a named team-level intervention | reporting needs redesign | no decision changes or evidence gap is clarified | | governance | permitted purpose, access, retention, and deletion operate | correct a bounded deviation | material unauthorized use or data exposure |
Do not use a pass-rate target as the primary gate. A high or low score distribution can reflect scenario difficulty, rubric design, population selection, access, or real capability differences.
- Access: whether a participant could enter, use required tools, and receive accommodations.
- Attempt: whether the assessment was started and completed under the intended conditions.
- Observed capability: evidence tied to the six AI fluency pillars and the specific task.
- Decision value: whether the report changed a supported programme decision or clarified an evidence gap.
O — Operate the simulation with safeguards
Run a technology check before the assessed session. Give participants a clear briefing, time expectation, support route, and explanation of what is observed. Log scenario version, assessment window, system incidents, accommodation delivery, scorer assignment, and any deviation.
Monitor operations without coaching participants toward desired answers. Separate technical support from assessment judgment. If the system fails, preserve the incident evidence and apply the predefined retry or incomplete-attempt rule. Do not silently exclude a failed attempt from the denominator.
Protect raw work evidence. Restrict access by purpose, record exports, set retention, and report aggregates at a level that does not expose individuals unnecessarily. Require human review of any AI-assisted scoring or report drafting and retain the evidence behind the reviewed result.
T — Test decision value, not a promotional claim
At pilot close, reconcile eligible, invited, started, completed, withdrawn, inaccessible, invalid, and scored counts. Present distributions with the scenario and rubric version, missingness, reviewer checks, and confidence limitations.
Ask five questions:
Report operational outcomes and capability evidence separately. Completion, participant satisfaction, or report delivery on time does not prove capability improvement. A later training or workflow change needs its own evaluation design.
- Did the assessment reveal a workflow-specific capability gap that other available evidence did not?
- Can a reviewer reconstruct material conclusions from observed process and deliverable evidence?
- Did access or selection patterns limit interpretation?
- Which intervention, workflow control, or reassessment decision changes because of the evidence?
- What would have to be true to expand the pilot, and what remains untested?
Worked pilot: customer-policy summary review
A synthetic service organisation wants to understand whether a customer-operations team can use AI to summarize a new returns policy while preserving exceptions and verifying claims. The CHRO is the capability-strategy owner and executive sponsor; customer operations owns workflow representativeness, information security reviews data handling, and learning operations owns follow-up support.
Forty-two employees meet the pre-registered eligibility rule. Thirty-six are selected using strata for two service groups and two shift patterns. Thirty-two complete the simulation, two encounter an access defect, and two withdraw. The two access failures remain in the operational denominator and are not scored as low capability. The scenario asks each participant to review a synthetic policy packet, use AI support if desired, resolve contradictory guidance, and deliver a customer-facing summary plus an exception log.
Double-scoring finds that one rubric criterion combines source fidelity with writing style. Reviewers disagree on six attempts because a polished summary omitted an exception while a less polished summary preserved it. The pilot pauses comparative reporting, splits the criterion for future use, and retains the original version for the current cohort. The evidence also shows a repeated failure to verify effective dates, while prompt length and tool-use frequency do not explain the difference.
The decision is to change and repeat a smaller pilot after repairing access and the criterion. Learning operations designs a source-verification exercise; customer operations adds an effective-date check to its live workflow. The organisation does not claim productivity gains, rank employees, or use the pilot for a hire/no-hire decision.
Copyable CHRO pilot charter
Two Prune can provide a realistic assessment simulation and evidence-backed report for this bounded pilot. It does not validate the organisation’s employment decisions, establish workforce outcomes, or replace qualified legal, privacy, security, accessibility, or psychometric review.
- Decision: the precise adoption or programme decision the pilot informs.
- Workflow: business deliverable, audience, constraints, and representative owner.
- Alternatives: existing evidence and the remaining gap.
- Population: eligibility, sampling, participation, accessibility, and missingness rules.
- Protocol: scenario, rubric, evidence layers, reviewer controls, and version freeze.
- Governance: purpose, consent or notice, access, retention, deletion, and incident route.
- Gates: continue, change, stop, and escalation conditions agreed before launch.
- Operations: owners, schedule, support, incidents, and deviation log.
- Closeout: reconciled counts, findings, limitations, decision, and next evidence gate.
Sources
- 1.CIPD guide to AI skills planning · www.cipd.org
- 2.U.S. Department of Labor AI best practices · www.dol.gov