Practical guide
AI capability pilot design for a legal leader
A legal leader should sponsor an AI-capability pilot only when it answers a bounded operational question with protected materials, accountable supervision, and pre-agreed evidence. The LEGAL method below produces a pilot protocol rather than a sales demonstration: Limit the use case, Establish the baseline, Gate risks and evidence, Assign accountable review, and Learn against decision rules. The.
A legal leader should sponsor an AI-capability pilot only when it answers a bounded operational question with protected materials, accountable supervision, and pre-agreed evidence. The LEGAL method below produces a pilot protocol rather than a sales demonstration: Limit the use case, Establish the baseline, Gate risks and evidence, Assign accountable review, and Learn against decision rules. The outcome may be proceed, repair and rerun, narrow the scope, or stop.
L — Limit the use case and decision
Choose one recurring legal-work task whose inputs, expected output, review standard, and failure consequences can be described. Examples include preparing a first-pass contract-risk summary from a fixed document set or drafting a chronology from approved matter files. Avoid a pilot framed as “use AI in legal” because neither capability nor risk can be interpreted at that level.
Write a one-sentence decision: “After the pilot, the legal leadership group will decide whether to extend, repair, or stop this workflow under the stated controls.” Name excluded uses, matter types, data classes, jurisdictions, tools, and participant groups. A capability pilot is not an approval of a product for every legal task.
E — Establish the baseline and alternatives
Measure the current workflow before introducing the simulation or tool. Record task volume, review time, common error categories, escalation frequency, and evidence already available from supervision. Include realistic alternatives: existing supervisor review, a knowledge check, guided practice, or a narrower process control.
The baseline should use the same task definition and quality rubric planned for the pilot. If the proposed pilot uses clean synthetic documents but the comparison uses unusually difficult live matters, the result will be uninterpretable. Document selection rules before seeing results and preserve a matter-independent record of exclusions.
G — Gate confidentiality, accuracy, and evidence
Create a risk-and-evidence register with one row per material failure. For each row record the trigger, preventive control, observable evidence, reviewer, escalation path, and stop condition. Legal-work examples include invented authorities, missed contract clauses, disclosure of confidential information, unsupported conclusions, loss of source traceability, and excessive deference to an AI output.
The SRA misuse-of-AI warning notice says existing professional obligations remain with regulated people and firms when AI is used. It highlights inaccurate information, confidentiality, human oversight, and a proportionate risk-based approach. Translate those duties into the pilot protocol; do not treat a tool setting or participant acknowledgement as sufficient evidence.
Before starting, confirm:
- only synthetic, anonymised, or specifically approved materials enter the exercise;
- source documents and expected issues are version controlled;
- the rubric separates issue spotting, source accuracy, uncertainty, escalation, and communication;
- reviewers are trained on the rubric and test-score a calibration example;
- telemetry and retained evidence are proportionate, disclosed, access controlled, and time bounded;
- an incident owner can pause the pilot without waiting for the final review.
Sources: [1]
A — Assign accountable human review
Name the legal owner, operational owner, privacy or information-governance reviewer, assessment designer, and independent quality reviewer. The same person may hold more than one role in a small team, but every responsibility must remain explicit.
The SRA effective-supervision guidance states that AI-assisted or AI-generated work should receive appropriate human review, scrutiny, and professional judgement, with an authorised individual retaining ultimate responsibility for legal services. For a capability pilot, supervision evidence should therefore include what the participant checked, what the reviewer checked, disagreements, corrections, and the final disposition—not merely whether a task was submitted.
Run a short calibration round before the scored sample. Reviewers independently score the same two artifacts, compare category-level differences, revise ambiguous anchors, and repeat if disagreement could change the decision. Preserve the original and reconciled scores so uncertainty is visible.
Sources: [2]
L — Learn against pre-agreed decision rules
Define the minimum evidence packet and decision rules before results are known. A packet might contain participant artifacts, source citations, revision history, rubric scores, reviewer notes, critical-error counts, escalation behaviour, completion time, missing-data log, and incident record. Analyse results at the workflow level; a small pilot does not establish general capability across legal practice.
Use rules such as:
The SRA’s AI Growth Lab description explicitly distinguishes responsible testing with regulatory input from approval, endorsement, or authorisation. Apply the same boundary internally: a successful pilot supports only the named next decision under the tested conditions.
- Extend: required evidence is complete, no stop condition occurred, critical-error handling meets the agreed threshold, and reviewers reach acceptable agreement.
- Repair and rerun: the workflow is promising but the rubric, materials, supervision, or telemetry prevented a sound interpretation.
- Narrow: evidence supports only a smaller task, data class, or participant group.
- Stop: confidentiality controls failed, critical errors escaped required review, or the proposed decision cannot be supported by the collected evidence.
Sources: [3]
Worked pilot protocol — contract-risk summaries
A legal team wants to know whether a realistic simulation can reveal how consistently people verify AI-assisted contract-risk summaries. It selects twelve volunteers from two practice groups and a synthetic contract pack containing a limitation-of-liability conflict, a stale governing-law reference, a hidden renewal term, and one irrelevant distractor. The baseline is an existing supervisor-reviewed exercise completed without AI support.
The pilot allows one approved AI tool with synthetic text only. Participants must submit a risk table containing the clause location, issue, consequence, confidence, proposed action, and evidence checked. Reviewers score source traceability, issue detection, unsupported additions, escalation, and clarity. A fabricated authority or unreported confidentiality incident is a stop condition. Reviewer disagreement of more than one rubric band on any critical category triggers recalibration rather than averaging.
Results show that nine artifacts trace every conclusion, two miss the renewal term, and one adds a nonexistent statutory requirement but flags it as unverified before submission. Reviewers disagree materially on whether that correction demonstrates adequate control. The leadership group chooses repair and rerun: it sharpens the anchor for corrected AI errors, adds a second calibration artifact, and keeps the use case bounded. It does not infer that the participants are capable across all legal work or that the tool is approved for live client matters.
Two Prune boundary
Two Prune can run realistic work simulations and provide evidence about observed participant performance across context framing, evidence navigation, AI orchestration, verification and risk control, judgement and synthesis, and communication. The customer defines legal scope, materials, authorised tools, supervision, privacy controls, and real-world decisions. Two Prune is not a law firm, regulator, product-approval service, or automated employment-decision system.
Sources
- 1.SRA misuse-of-AI warning notice · www.sra.org.uk
- 2.SRA effective-supervision guidance · www.sra.org.uk
- 3.SRA’s AI Growth Lab description · www.sra.org.uk