Practical guide
AI capability pilot design for a future-of-work researcher
A future-of-work research pilot should answer a live workforce decision with the smallest defensible study. It should not begin by trying to produce an organization-wide capability score. The researcher is often the owner of construct definition, study design, analysis, and limitations. A programme leader or business sponsor may own the investment decision. Governance, workforce analytics,.
A future-of-work research pilot should answer a live workforce decision with the smallest defensible study. It should not begin by trying to produce an organization-wide capability score. The researcher is often the owner of construct definition, study design, analysis, and limitations. A programme leader or business sponsor may own the investment decision. Governance, workforce analytics, learning, employee representatives, and functional experts may act as reviewers or collaborators. Make those roles explicit before recruitment. Use the PULSE pilot method: Purpose, Units, Learning contrast, Signals, and Exit.
Purpose: bind the pilot to one decision
Write the decision and alternatives in one paragraph:
> By [date], [owner] will decide whether to continue the current approach, repair a workflow, test a targeted practice intervention, expand access, or stop. The pilot will provide evidence about [construct] for [population] performing [task] under [conditions].
Then state what the pilot cannot answer. Common exclusions include productivity across all work, transfer to different roles, long-term behavior change, causal programme impact without a credible comparison, and individual employment suitability.
The business-case phase identifies whether an evidence gap matters. The pilot phase turns that gap into a study with participants, tasks, measures, operating controls, and exit rules.
Units: define population, task, and evidence unit
### Population
Specify the eligible group, recruitment route, expected sample, non-participation, and important underrepresentation. Record whether participation is voluntary, expected, or part of an existing learning activity. Avoid interpreting a convenience sample as the whole workforce.
### Task
Choose a task that creates a fair opportunity to observe the capability of interest. A useful simulation includes a clear audience and deliverable, relevant source material, ambiguity, a designed evidence challenge, enough time to work, and a policy-consistent way to use AI.
### Evidence unit
Define the unit before collection. It might be one completed session, one task attempt, one reviewed deliverable, or one participant across two alternate forms. Keep session, participant, reviewer, and task identifiers distinct so repeat attempts and missing data are not silently merged.
OECD employer and worker surveys deliberately combine different viewpoints: employer data describes implementation, while worker data captures first-person experience. A pilot can borrow that complementarity by pairing observed task evidence with participant and manager context, while preserving the distinction between perception and performance. OECD: AI surveys of employers and workers
Sources: [1]
Learning contrast: choose what will change the decision
A contrast is more useful than a vague before-and-after story. Select one design that the available sample and operating conditions can support:
Predefine allocation, timing, alternate-form differences, contamination risks, and which questions remain descriptive. If assignment is not random, preserve likely selection effects in the interpretation.
- baseline and targeted practice: comparable task forms before and after a defined intervention;
- support comparison: two groups receive different forms of guidance or workflow support;
- designed challenge: every participant encounters the same contradiction, missing source, or verification trap;
- replicated case: two functions complete equivalent tasks to test whether the method travels;
- single-cohort feasibility: no effect claim, but strong evidence on task operation, reviewer consistency, missingness, and burden.
Signals: collect a layered evidence packet
Separate the evidence layers.
| Layer | Example | Main limitation | | --- | --- | --- | | Context | role family, prior exposure, task familiarity | self-report or administrative quality | | Process | source review, AI iteration, checks, revisions | only observable events are captured | | Output | deliverable accuracy, usefulness, traceability | can hide unseen reasoning | | Reviewer | rubric scores, rationale, confidence, disagreement | depends on task and training | | Operation | missing events, support time, review effort, incidents | small pilot conditions may not scale | | Experience | clarity, burden, perceived relevance | perception is not task performance |
Define every measure with a construct, source, denominator, time window, missingness rule, and interpretation boundary. Do not let tool usage stand in for applied capability.
NIST's AI RMF Core recommends defining the tasks and methods an AI system supports, documenting test methods and results, considering uncertainty, and evaluating in conditions resembling deployment. A workforce researcher can use that discipline for the pilot's evidence design without claiming that the pilot evaluates the AI system or establishes a universal benchmark. NIST AI RMF Core
Sources: [2]
Exit: pre-register continue, repair, hold, and stop rules
Set the rules before seeing results.
### Continue
### Repair
### Hold
### Stop
- the task elicits the intended behaviors;
- data capture and review are complete enough for the decision;
- no critical participant or operating issue remains open;
- the next stage answers a live question.
- task instructions, materials, rubric anchors, reviewer training, or capture can be corrected;
- the problem does not invalidate all collected evidence;
- a challenge retest can show whether the repair works.
- privacy, records, access, fairness, or participant communication needs review;
- missingness or reviewer disagreement prevents the planned interpretation;
- the sponsor has not resolved how results will be used.
- the decision no longer matters;
- the study cannot create a fair opportunity to demonstrate the construct;
- required evidence cannot be collected responsibly;
- the planned use exceeds the approved boundary.
Pilot protocol template
### Decision and roles
| Field | Entry | | --- | --- | | Decision and date | | | Decision owner | | | Research owner | | | Governance reviewer | | | Functional collaborator | | | Participant representative | |
### Design
| Field | Entry | | --- | --- | | Population and recruitment | | | Task and alternate forms | | | Contrast or designed challenge | | | Sample and allocation | | | Evidence layers | | | Missingness and exclusion rules | | | Reviewer training and adjudication | | | Data access, retention, and deletion | | | Exit gates | |
### Reporting boundary
Report the population, task, dates, completion, missingness, reviewer agreement, observed examples, uncertainty, operational burden, deviations, and the decision reached. Separate descriptive findings, comparisons, and hypotheses. Do not report a precision that the sample or rubric cannot support.
Worked pilot: source verification after an AI academy
A workforce research team must advise whether an operations AI academy should add a source-verification practice module. Usage data shows broad adoption, and participant surveys show high confidence, but neither source reveals how people reconcile conflicting policy evidence.
The pilot recruits 36 volunteers from two operations teams. The decision owner is the AI enablement director. The future-of-work researcher owns design and synthesis; the operations lead supplies task expertise; governance reviews data and use; learning owns any later intervention.
Each participant receives a synthetic exception-reporting task with a spreadsheet, two policy extracts, and an inbox clarification. One policy extract contains a later exception that changes the recommendation. Participants may use an approved AI assistant and must submit a decision memo with source anchors.
The study uses a designed challenge rather than an outcome comparison. Signals include documents opened, AI interaction where approved, verification actions, final source traceability, reviewer rationale, confidence, missing events, completion time, and participant feedback. Reviewers score independently, then adjudicate disagreements without changing the frozen rubric.
Pre-registered rules include:
The synthetic result is 33 complete sessions. Reviewers find that 19 participants cite the general policy without reconciling the later exception, while 11 identify and explain the conflict and 3 have insufficient evidence for interpretation. Reviewer agreement is acceptable after two calibration cases, but the event stream misses one offline calculation pattern.
The recommendation is repair and continue: fix capture guidance for offline calculations, then test a targeted source-conflict practice module with alternate task forms. The result is limited to this sample, task, and period. It does not show that the academy caused the observed pattern or that the result transfers to other work.
- repair if more than 10% of sessions lack critical event evidence;
- repair if reviewers cannot apply the source-conflict anchor consistently after training;
- hold if participants reasonably misunderstand how their evidence will be used;
- continue to a practice-module test only if the challenge distinguishes observable verification approaches and operating burden is acceptable to the owner.
Pilot failure traps
Recruiting only enthusiasts. Record selection and avoid population claims.
Measuring use instead of capability. Connect every signal to the defined construct.
Changing the rubric after seeing results. Freeze anchors and record deviations.
Treating missing evidence as poor performance. Use an insufficient-evidence disposition.
Letting one aggregate hide a critical failure. Preserve error types and examples.
Scaling before the method works. Require task, capture, review, and operating gates before expansion.
Making the researcher the default buyer. Assign owner, influencer, reviewer, and collaborator based on real authority.
Two Prune can provide a structured simulation and evidence packet for a bounded pilot. It does not prove workforce outcomes, replace a research design, establish mature norms, or make employment decisions. The pilot owner remains responsible for use, interpretation, participant communication, and the next decision.
Sources
OECD: AI surveys of employers and workers
- OECD: AI measurement in ICT usage surveys
- NIST AI Risk Management Framework Core
- U.S. GAO: Artificial Intelligence Accountability Framework
Sources: [1] [3] [2] [4]
Sources
- 1.OECD: AI surveys of employers and workers · www.oecd.org
- 2.NIST AI RMF Core · National Institute of Standards and Technology
- 3.OECD: AI measurement in ICT usage surveys · www.oecd.org
- 4.U.S. GAO: Artificial Intelligence Accountability Framework · U.S. Government Accountability Office