Practical guide

AI capability pilot design for a functional practice leader

A functional practice leader should design an AI capability pilot around one practice decision, not around a target number of participants or a desire to demonstrate that a platform works. The pilot should create comparable, task-bounded evidence, expose its limitations, and end with a predeclared decision. It should not rank participants, predict workplace performance, or treat tool usage as.

By Two Prune

A functional practice leader should design an AI capability pilot around one practice decision, not around a target number of participants or a desire to demonstrate that a platform works. The pilot should create comparable, task-bounded evidence, expose its limitations, and end with a predeclared decision. It should not rank participants, predict workplace performance, or treat tool usage as proof of capability. The practice leader usually owns the work standard and reviews whether evidence is useful. They may influence funding, but sponsor and budget authority must be mapped rather than assumed. Learning, technology, security, accessibility, legal, and participant managers may be collaborators or reviewers.

Use the FIELD pilot method

FIELD keeps the pilot small enough to interpret and specific enough to change a real practice decision.

### F: Freeze the practice decision

Write one question the pilot must inform. Examples include whether a verification standard needs targeted practice, whether a new workflow is ready for wider testing, or whether a common simulation adds useful evidence beyond current manager observation.

Define the practice standard in observable terms. “Use AI well” is not sufficient. A sourced recommendation task might require the participant to identify material constraints, distinguish source facts from inference, trace consequential claims, resolve or escalate contradictions, and communicate residual uncertainty.

Record the current evidence and its limitations. Usage analytics can show activity. Course records can show participation. Manager observation can provide contextual judgment. Live work review can show real outputs. None automatically creates a consistent opportunity for several people to handle the same task, evidence, time limit, and ambiguity.

### I: Identify authority, participants, and protections

Map the practice owner, sponsor, budget authority, pilot operator, evidence reviewer, participant manager, accessibility reviewer, security or privacy reviewer, and stop authority. Label each as owner, influencer, reviewer, or collaborator.

Set a transparent inclusion rule. A convenience sample may be acceptable for an exploratory test, but it limits interpretation. State whether participation is voluntary, how accommodations are handled, what data is collected, who can view it, how long it is retained, and how results will and will not be used.

UK Government functional-standard guidance is written for government organizations and is not a universal private-sector model. It is useful here because it emphasizes proportionate application, defined roles and accountabilities, governance, assurance, and continuous improvement. Apply those ideas to the organization’s context rather than copying the structure. Implementing functional standards: a guide for senior leaders

### E: Engineer the evidence opportunity

Build a realistic synthetic or explicitly approved non-sensitive task that represents a consequential part of the practice. Give every participant the same core outcome, evidence environment, constraints, time boundary, and available tools. Include ambiguity, competing evidence, and at least one verification trap that is relevant to the practice standard.

Freeze the assignment, source materials, expected deliverable, observation plan, scoring or interpretation guide, and missing-evidence rules. Decide which workflow signals are actually available. Do not infer that an unobserved check did not occur, and do not treat a tool event as proof that the participant understood or verified the result.

The evidence packet should include the submitted deliverable, relevant available workflow evidence, reviewer annotations, confidence limitations, unresolved disagreements, and follow-up questions. The packet is for human interpretation, not an automated employment recommendation.

### L: Lock the interpretation and learning plan

Train reviewers on the evidence protocol before results are visible. Give them representative examples and require them to point to evidence for each judgment. Use double review on a small calibration subset when feasible. Record disagreements about evidence location separately from disagreements about quality.

Predeclare the result-use rules:

Do not rewrite success criteria after seeing results. An interesting report is not automatically a useful pilot.

### D: Decide, document, and expire

Hold a decision meeting with the named authority. Review evidence quality, participant experience, reviewer burden, accessibility findings, security or privacy issues, and whether the result changed or sharpened the predeclared decision. Separate what the pilot observed from what it cannot establish about transfer to live work.

Record the disposition, owner, next evidence gate, and expiry date. A decision to run another bounded cohort is not authorization for enterprise rollout. Scenario, source, model, participant-population, or interpretation changes should trigger targeted review.

  • proceed if the evidence is interpretable, proportionate, and useful for the named practice decision
  • repair if the task, sources, instructions, accessibility, or reviewer protocol caused avoidable ambiguity
  • hold if missing evidence prevents a defensible interpretation
  • stop if data handling, participant impact, or result use crosses the approved boundary

Sources: [1]

Copyable pilot protocol

| Field | Required entry | | --- | --- | | practice decision | one question the evidence must inform | | observable standard | behaviors and deliverable requirements | | current evidence gap | what present methods cannot resolve | | participant rule | inclusion, exclusions, accommodations, consent route | | task boundary | synthetic or approved materials, time, tools, prohibited actions | | evidence packet | deliverable, available workflow evidence, annotations, missingness | | reviewers | role, training, calibration subset, conflict route | | protections | access, retention, privacy, security, accessibility, stop authority | | disposition rules | proceed, repair, hold, stop | | result use | named practice decision and prohibited uses | | expiry | date and evidence needed for the next decision |

Worked example: proposal-quality practice

A proposal practice leader sees an increase in unsupported statements and missed mandatory requirements in quality-review notes. Tool usage also increased, but deadlines, work mix, source quality, and reviewer behavior changed. The leader cannot attribute the defects to AI.

Freeze: the pilot asks whether a consistent red-team simulation produces useful evidence for choosing the next verification-practice exercise. The observable standard requires participants to identify mandatory requirements, link corrections to supplied evidence, flag unsupported statements, and communicate a prioritized correction plan.

Identify: the practice leader owns the standard and decision. A learning partner operates the session, a security reviewer approves synthetic materials, an accessibility reviewer checks the task, and two trained reviewers interpret the evidence. A separate sponsor controls spend. Results cannot be used to rank participants or make employment decisions.

Engineer: twelve volunteers complete the same 30-minute synthetic task. The evidence set contains a conflicting requirement and a plausible unsupported claim. The assessment captures the final correction plan and available workflow evidence. Missing events are labeled unavailable rather than scored as failures.

Lock: reviewers use a frozen protocol. They double-review four submissions and record evidence-location agreement, interpretation time, uncertainty, and follow-up questions. Proceed requires that managers can use the evidence for the named practice decision without disproportionate burden. A task ambiguity, material accessibility barrier, or unresolved data-handling issue triggers repair or stop.

Decide: the evidence consistently exposes missed source conflicts but one instruction caused reviewers to disagree about expected escalation. The disposition is repair and repeat a small test. The leader fixes that instruction and retains the original result rather than declaring success. The pilot has informed scenario repair and a practice question, but it has not proven workplace transfer or business impact.

Two Prune creates controlled work simulations and structured assessment evidence for human interpretation. It does not prove future performance, promise business outcomes, rank participants, or make automated employment decisions.

Sources

UK Cabinet Office: Implementing functional standards, a guide for senior leaders

  • UK Cabinet Office: Guide to functional standards

Sources: [1] [2]