Practical guide

AI capability pilot design for a people analytics leader

A people analytics leader can help design an applied AI capability pilot when the organization needs evidence for a bounded people or programme decision. The role is usually an evidence adviser, measurement partner, or reviewer rather than the automatic owner of training, technology, budget, privacy, or employment decisions. A sound pilot makes those local decision rights explicit. Use the PILOT.

By Two Prune

A people analytics leader can help design an applied AI capability pilot when the organization needs evidence for a bounded people or programme decision. The role is usually an evidence adviser, measurement partner, or reviewer rather than the automatic owner of training, technology, budget, privacy, or employment decisions. A sound pilot makes those local decision rights explicit. Use the PILOT method: purpose, inclusion, local evidence, observation protocol, and transition decision.

Source basis and limits

CIPD describes people analytics as using people and business data to inform organizational decisions and places organizational data alongside professional expertise, scientific literature, and stakeholder perspectives. The UK AI Skills Tools Package asks employers to learn from early experiments or pilots, identify workforce gaps, and plan upskilling. The current employer guide recommends training connected to real roles, workflows, systems, governance, responsible practice, and follow-up review. These sources support a mixed-evidence pilot; they do not establish that a simulation measures workplace transfer, prove business impact, or authorize a people analytics leader to repurpose participant evidence. CIPD people analytics guide CIPD evidence-based practice AI Skills Tools Package Employer guide to AI upskilling

Sources: [1] [2] [3] [4]

P — Purpose and next decision

Begin with one decision the pilot can inform. Examples include whether to repair access before training, whether a role-specific verification module is warranted, or whether a larger evaluation is justified. Avoid broad purposes such as “measure workforce readiness.”

Write a pilot charter with:

A simulation can create a consistent opportunity to observe task-bounded behavior. It cannot by itself prove future workplace performance, productivity, learning transfer, or organizational return.

  • the workflow, task, population, and operating context;
  • the accountable owner of the next decision;
  • the people analytics role: owner, influencer, reviewer, or collaborator;
  • the evidence gap and current alternatives;
  • the smallest useful pilot question;
  • what the pilot can and cannot establish;
  • the stop date and decision date; and
  • prohibited secondary uses of participant evidence.

I — Inclusion, access, and rights

Define who is eligible and how people are invited. Record role coverage, workload constraints, language, disability access, approved tools, prior exposure, and any support needed. Participation and analysis rules should be decided before results are known.

Use an inclusion ledger:

| Participant state | Operational meaning | Analysis treatment | Repair route | | --- | --- | --- | --- | | Eligible and accessible | Fair opportunity under protocol | Include with evidence-quality checks | N/A | | Access or accommodation failure | Opportunity was not provided as designed | Report unavailable, not low performance | Repair and repeat | | Missing critical evidence | Construct cannot be judged confidently | Apply predefined missingness rule | Review instrumentation | | Protocol deviation | Conditions differ materially | Separate or exclude as specified | Investigate cause |

Specify purpose limitation, access control, retention, correction, review, and deletion routes under applicable organizational and legal requirements. Do not silently convert a development pilot into screening, ranking, or a live employment decision.

L — Local evidence and rival explanations

Assemble evidence about the actual workflow before designing the task:

Create an evidence map:

| Question | Existing evidence | Gap | Pilot evidence | Rival explanation | Interpretation limit | | --- | --- | --- | --- | --- | --- | | | | | | | |

Low usage is not proof of low capability. High confidence is not proof of reliable work. A poor artifact can reflect task ambiguity, inaccessible materials, missing domain knowledge, time pressure, or weak verification. The pilot should preserve these distinctions.

  • representative work products and quality criteria;
  • approved sources, tools, policies, and escalation routes;
  • known access, policy, manager-support, or workflow constraints;
  • existing survey, learning, usage, quality, and incident evidence;
  • stakeholder views from affected workers and specialists; and
  • alternative explanations for the observed problem.

O — Observation and analysis protocol

Design one representative, synthetic or explicitly approved non-sensitive task. Give every participant a comparable opportunity to demonstrate the target behaviors. Predefine:

Do not produce a single score without retaining the evidence needed to interpret it. Separate eligibility, access, self-report, observed task performance, artifact quality, workflow evidence, and later operational outcomes.

Use an analysis table:

| Measure | Construct | Source | Denominator | Missingness rule | Critical exception | Confidence | | --- | --- | --- | --- | --- | --- | --- | | | | | | | | |

Run a dry test with reviewers who did not author the task. Check whether the evidence actually creates an opportunity to observe each behavior and whether the rubric distinguishes missing evidence from poor performance.

  • task inputs, ambiguity, constraints, time limit, and expected deliverable;
  • observable behaviors and artifact criteria;
  • critical failures that cannot be averaged away;
  • evidence sources, instrumentation, and missingness rules;
  • reviewer training, independence, and disagreement process;
  • subgroup and accessibility checks where lawful and appropriate;
  • AI-use rules, data boundaries, and verification expectations; and
  • analysis outputs and confidence labels.

T — Transition decision and evidence gates

Precommit to dispositions rather than declaring success after any completed pilot:

The transition record should name the decision owner, evidence reviewed, adverse findings, dissent, uncertainty, action, prohibited inference, and review date. Completion is not authority to scale.

  • stop: the purpose is inappropriate, rights cannot be protected, or the task lacks relevance;
  • repair: access, task design, instrumentation, reviewer consistency, or evidence completeness is inadequate;
  • repeat: the design is usable but uncertainty remains too high for the next decision;
  • proceed narrowly: the evidence supports one bounded learning or workflow action; or
  • evaluate later: a separate study examines transfer or operational outcomes.

Worked pilot: fictional complaint-response team

A people analytics leader is asked to test whether a 30-person “AI readiness” pilot should guide a company-wide training purchase. PILOT narrows the next decision: should the customer-operations team receive a verification module for evidence-based responses to complex complaints?

Existing quality reviews show unsupported explanations and inconsistent source use, but usage data cannot distinguish access, policy, knowledge, or verification practice. The people analytics leader acts as measurement adviser. The service owner owns workflow standards, learning owns the response, privacy and employee-relations specialists review rights, and an executive sponsor owns any later investment.

The task uses synthetic complaint files, an approved reference library, a timed deliverable, and observable source-review and AI-verification evidence. The protocol predefines unsupported citations and fabricated policy claims as critical exceptions. It also records access and evidence completeness separately.

During delivery, six people cannot open the approved reference system. Their results are marked unavailable and routed for repair, not scored as poor capability. Among 24 accessible sessions, three final responses contain unsupported citations and several participants notice but fail to correct contradictory policy dates. Reviewers agree on the critical exceptions but disagree on two judgment ratings, so the confidence note preserves that disagreement.

The decision owner approves a targeted verification module and a repeat pilot after access repair. The record explicitly withholds claims about workforce readiness, productivity, transfer, or return. Company-wide expansion remains unapproved pending the repeat and separate workplace-quality evidence.

Pilot quality checklist

[ ] One bounded next decision and accountable owner are named.

  • [ ] The people analytics role and other decision rights reflect local authority.
  • [ ] Eligibility, access, accommodation, consent or notice, retention, and correction routes are defined.
  • [ ] The task represents a real workflow without using unapproved sensitive material.
  • [ ] Observable behaviors, artifact criteria, critical failures, and missingness rules are predeclared.
  • [ ] Reviewers are trained and disagreement is retained.
  • [ ] Self-report, usage, observed capability, work quality, and outcomes remain separate.
  • [ ] Stop, repair, repeat, proceed, and later-evaluation gates are explicit.
  • [ ] Prohibited inferences and secondary uses are recorded.

Two Prune boundary

Two Prune can support a bounded assessment pilot by providing a realistic simulation, comparable task conditions, workflow and deliverable evidence, and a structured report for human interpretation. It does not establish workplace transfer, validate organizational outcomes, determine employment decisions, replace legal or specialist review, or give a people analytics leader authority they do not hold locally.

Sources

CIPD: People analytics guide

  • CIPD Profession Map: Evidence-based practice
  • GOV.UK: AI skills tools package
  • GOV.UK: Employer guide to AI upskilling

Sources: [1] [2] [3] [4]

Sources

  1. 1.CIPD people analytics guide · prod.cipd.org
  2. 2.CIPD evidence-based practice · www.cipd.org
  3. 3.AI Skills Tools Package · UK Government
  4. 4.Employer guide to AI upskilling · UK Government