Practical guide

How an organizational development leader can design an AI capability pilot

An organizational development leader should treat an AI capability pilot as a small work-system experiment, not a software roadshow or a short course with a satisfaction survey. The pilot should test whether people can perform one bounded workflow with better evidence, clearer controls, and a workable division of responsibility. Its output is a decision packet about the next intervention, not a.

By Two Prune

An organizational development leader should treat an AI capability pilot as a small work-system experiment, not a software roadshow or a short course with a satisfaction survey. The pilot should test whether people can perform one bounded workflow with better evidence, clearer controls, and a workable division of responsibility. Its output is a decision packet about the next intervention, not a claim that the organization has become AI capable.

The stakeholder's role

For this pilot, the organizational development leader is the design owner for the work-system hypothesis, participant conditions, learning loops, and interpretation. The workflow owner remains accountable for operational quality. Information-security, privacy, legal, accessibility, and employee-representative specialists are reviewers where their mandates apply. Learning, data, technology, and frontline managers are collaborators. An executive sponsor may own funding and the decision to extend the pilot. Do not assign every responsibility or budget decision to the organizational development leader.

CIPD describes organization development and design as work concerned with organizational effectiveness and health, informed by systems thinking, organization development theory, and behavioral science. NIST's AI RMF says risk management should be continuous, lifecycle-based, and multidisciplinary. The UK government's AI upskilling guide emphasizes practical work, integration with organizational needs, modular access, continued support, and transfer into the workplace. Together, these sources support a pilot that changes and observes a work system rather than measuring attendance alone. CIPD Profession Map NIST AI RMF Core UK Employer Guide

Sources: [1] [2] [3]

Use the SHIFT pilot protocol

SHIFT separates five decisions that pilots often blur: select the work, hypothesize the change, instrument the evidence, fix the safeguards, and translate the result.

### S — Select one decision-bearing workflow

Choose a workflow with an identifiable customer, output, evidence base, owner, and review point. It should occur often enough to observe several cases but remain reversible during the pilot. Write down:

Avoid a topic such as “use generative AI more.” A suitable pilot question is narrower: “Can analysts produce traceable first-draft policy comparison tables from an approved document set while keeping source conflicts visible?”

### H — Hypothesize the work-system change

Describe the proposed change as an observable mechanism:

> If trained analysts use a source-linked comparison procedure, an approved AI tool, and an independent release check on bounded cases, then they should produce usable first drafts with fewer unsupported cells and no increase in unresolved sensitive-data events, because the procedure makes evidence and escalation explicit.

Specify what changes in task sequence, role, tool, information, control, and feedback. State competing explanations. Faster drafts may result from simpler pilot cases; better quality may come from extra reviewer attention rather than the AI-supported procedure. A credible design plans how to distinguish these possibilities.

### I — Instrument evidence before launch

Use a small evidence map rather than one success metric.

| Decision question | Measure | Evidence source | Comparison | Interpretation boundary | | --- | --- | --- | --- | --- | | Was the output usable? | material corrections and release disposition | blinded rubric review | matched current-process cases | reviewer judgment is not error-free | | Was evidence handled well? | supported cells, unresolved conflicts, source retrieval accuracy | final table and source log | baseline rate | case difficulty must be tagged | | Did the workflow remain controlled? | sensitive-data events, bypasses, escalations, stop-rule activations | event log and incident record | zero-tolerance or policy threshold | absence of reports is not proof of absence | | Was effort redistributed? | active preparation, verification, and reviewer minutes | time log and workflow events | matched median | do not treat elapsed time as productive time | | Can the behavior transfer? | performance on a novel case after support is reduced | delayed assessment artifact | predefined rubric | one task does not establish broad capability |

Freeze case selection and review rules before viewing results. Use the same output standard for comparison cases. Where random assignment is impractical, match cases by difficulty and run them in alternating order. Keep participant identifiers out of the decision packet unless they are genuinely required and lawfully handled.

### F — Fix safeguards and stop rules

Name the approved data, tools, access rules, human checkpoints, retention period, and incident route. Include a no-fault escalation path so participants do not hide uncertainty to protect a pilot metric. Set stop rules before launch, such as:

A stop is evidence about the design. It should trigger containment, diagnosis, and a decision to repair, narrow, or end the experiment—not pressure to continue for a cleaner result.

### T — Translate findings into the next decision

Report the result by mechanism, case type, and condition. Separate observed evidence from interpretation. Use one of four dispositions:

Record what remains unknown. A successful policy-comparison pilot does not establish capability for customer communication, coding, financial analysis, or unrestricted research.

  • the current trigger, steps, tools, handoffs, output, and failure costs;
  • who makes which decisions and who is affected;
  • case types, including ordinary, ambiguous, sensitive, and failure cases;
  • the existing standard for an acceptable output; and
  • exclusions that keep the experiment bounded.
  • unapproved personal or confidential data enters the tool;
  • an output is released without the required source check;
  • a material conflict is suppressed or an evidence link is fabricated;
  • access, logging, or deletion controls fail; or
  • workload or accessibility conditions make participation unsafe or exclusionary.
  • stop because a critical condition cannot be controlled;
  • repair and repeat because the mechanism is plausible but the design failed;
  • extend cautiously to a named workflow or population with the same controls; or
  • adopt locally and monitor because the bounded workflow met its release criteria.

Worked pilot: a policy comparison workflow

A fictional public-services organization asks 12 policy analysts to compare four internal policy options against a controlled pack of 18 documents. Six analysts use the current spreadsheet process on one matched case and the SHIFT procedure on another; the order alternates. The pilot lasts three weeks. The organizational development leader owns the experiment design, while the policy director owns output quality and release. Security and records specialists approve the tool and retention boundary.

The new procedure requires each comparison cell to contain a source ID, page location, evidence status, and conflict note. Analysts can use AI to propose a first draft only from the approved pack. A second analyst checks decision-driving cells. Cases include one normal pack, one pack with conflicting effective dates, and one with an irrelevant but persuasive executive summary.

The current process produces a median of 11 material corrections per table; the pilot procedure produces six. Unsupported decision-driving cells fall from 17 across the comparison cases to five. Active preparation time falls by 19 minutes at the median, but verification time rises by 14 minutes. The normal cases improve most. In two conflict cases, the AI draft chooses the newer-looking date rather than preserving the disagreement. The required conflict field lets reviewers catch both before release.

No sensitive-data stop rule fires. One analyst bypasses the source-ID field by pasting a paragraph into a notes cell; the release check catches it. A delayed novel case shows that eight participants use the conflict field without prompting, while four need a reminder. These observations do not show organization-wide capability or isolate every cause: the reviewers knew they were in a pilot, and the sample is small.

The disposition is repair and repeat. The team adds a conflict-detection practice case, makes the source-ID field structurally required, reduces live coaching, and repeats on six new cases. The decision packet reports material corrections, source handling, effort distribution, control events, transfer evidence, missing observations, and the exact conditions under which any extension would occur.

Common objections

“Twelve participants are too few.” The pilot is not estimating a workforce prevalence. It is testing whether a mechanism and control set are observable enough to merit a larger study. Keep claims at that level.

“The assessment will slow people down.” Verification can add time while preventing rework or unsupported release. Report preparation and verification separately, and let the workflow owner decide the tradeoff.

“People will feel monitored.” Collect only evidence required for the pilot decision, explain the purpose and retention, involve appropriate representatives, and avoid repurposing the data for individual employment decisions.

Two Prune boundary

Two Prune can provide a controlled assessment in which participants complete realistic work, use AI support, navigate evidence, verify outputs, and submit a business deliverable. It can report observed assessment performance and evidence strength. It does not prove workplace productivity, diagnose an organization, certify broad AI capability, replace a workplace pilot, or make employment decisions.

Sources

CIPD: Organisation development and design

  • UK Government: Employer guide—what works for AI upskilling
  • NIST: AI Risk Management Framework Core

Sources: [1] [3] [2]

Sources

  1. 1.CIPD Profession Map · www.cipd.org
  2. 2.NIST AI RMF Core · National Institute of Standards and Technology
  3. 3.UK Employer Guide · UK Government