Practical guide
How a legal leader can build the case for applied AI capability evidence
A legal leader should build the case for applied AI capability evidence around a specific professional-risk and operating decision, not around a promise that assessment will improve legal outcomes. The legal leader is usually an adviser, reviewer, defined control owner, or collaborator; business, learning, technology, privacy, and security owners keep their own authority. Use PROVE: pin the.
A legal leader should build the case for applied AI capability evidence around a specific professional-risk and operating decision, not around a promise that assessment will improve legal outcomes. The legal leader is usually an adviser, reviewer, defined control owner, or collaborator; business, learning, technology, privacy, and security owners keep their own authority. Use PROVE: pin the decision, review alternatives, own the evidence and risk map, verify through a bounded pilot, and evaluate whether the evidence changes the decision. The artifact is an option-appraisal memo with a pilot gate, not an ROI forecast.
P: Pin the decision and the legal leader's role
State the decision in operational terms. Examples include whether to add realistic practice after AI training, whether a supervision process needs observable evidence, or whether a defined team should pilot an assessment before broader use. Name the accountable decision-maker and classify the legal leader's role for each issue.
Record the work in scope: task, jurisdiction, information type, external audience, consequence of error, professional responsibility, and current control. Separate evidence about access, course completion, and self-confidence from evidence about how someone handles a realistic AI-assisted task.
The Solicitors Regulation Authority's 2026 warning notice describes concerns involving inaccurate AI-generated legal work and confidential information placed in unsuitable tools. Its supervision guidance says AI-assisted legal work requires appropriate human review, scrutiny, and professional judgment, with an authorised person retaining ultimate responsibility. These sources support careful task design and supervision; they do not show that one assessment product produces better legal services.
R: Review the real alternatives
A credible business case compares alternatives rather than presenting a preferred product as inevitable. Include options such as:
For each option, compare decision usefulness, evidence quality, confidentiality, burden, accessibility, realism, consistency, administration, review requirements, and failure modes. A manager observation may be close to real work but inconsistent across supervisors. A simulation may standardise conditions but cannot reproduce every client, matter, or jurisdiction. Work samples may be authentic yet difficult to compare and unsafe to centralise.
Include the status quo as a real option. If existing supervision already answers the decision reliably and proportionately, a new assessment may add burden without changing action.
- retain current supervision and incident evidence;
- improve policy and training without a new assessment;
- add manager observation with a structured rubric;
- run a bounded simulation pilot;
- use reviewed work samples where confidentiality and comparability permit;
- combine several evidence sources;
- defer the decision while fixing an upstream process.
O: Own the evidence, cost, and risk map
Build a map from decision to evidence rather than a list of features. For each claim, record the observation needed, source, owner, missingness rule, interpretation limit, and action. Capability evidence might include whether a participant protects sensitive information, checks primary authority, detects a fabricated citation, reconciles conflicting instructions, escalates uncertainty, and produces a bounded deliverable.
Map costs without inventing a financial return. Include scenario design, synthetic materials, accessibility review, participant time, supervisor or reviewer time, administration, security and privacy review, integration, support, remediation, and periodic refresh. Identify which costs are known, estimated, variable, or excluded.
Map risks and controls separately:
The SRA's AI legal-market risk outlook recommends testing, assurance, quality control, and ongoing supervision. Use that guidance to define relevant safeguards, while recognising its jurisdiction and date.
- live or privileged information entering an assessment;
- a scenario that tests trivia instead of the intended capability;
- incomplete event capture treated as participant failure;
- scores used outside their supported context;
- small or unrepresentative comparison groups;
- inaccessible interaction or time pressure unrelated to the task;
- legal ownership expanded to decisions held elsewhere;
- assessment evidence converted into an automated employment recommendation.
V: Verify the proposition through a bounded pilot
Pre-register a pilot before seeing results. Specify the decision, representative task, eligible group, comparison or baseline, evidence fields, reviewer protocol, accessibility checks, missing-data rules, incident route, stop conditions, and dispositions. Do not define success as “participants completed the assessment.”
Useful pilot questions include:
Possible dispositions are stop, repair and rerun, continue the bounded pilot, or proceed to a separately governed expansion. A pilot is not an automatic path to scale.
- Does the task elicit the evidence the decision requires?
- Can reviewers locate the supporting evidence consistently?
- Are missing events distinguished from observed omissions?
- Does the report change a supervision, practice, or learning decision?
- What burden falls on participants, reviewers, and operators?
- Which groups or task variants are not represented?
- What must be repaired before another run?
E: Evaluate decision value, not promotional metrics
At the end, compare the pilot evidence with the original alternatives. Ask whether it reveals information that current evidence misses, whether that information is credible enough to act on, and whether the action is proportionate. Report disagreements, exclusions, missingness, and operational failures alongside any useful evidence.
Avoid using activation, completion, satisfaction, or prompt volume as proof of applied capability. Avoid describing a score as general legal competence. Do not convert a synthetic task into a claim about future client work. If the evidence does not change a decision, that is a material result and may support stopping or redesigning the approach.
Option-appraisal memo
| Field | Required content | |---|---| | Decision | Exact choice, owner, date, and scope | | Legal role | Owner, adviser, reviewer, or collaborator by issue | | Current evidence | What exists, its gaps, and its decision use | | Alternatives | Status quo and feasible non-product options | | Evidence needed | Observable behaviour, deliverable evidence, and missingness | | Cost map | Known, estimated, variable, and excluded costs | | Risk map | Failure, consequence, control, and escalation owner | | Pilot | Task, cohort, review, accessibility, and stop rules | | Decision gates | Stop, repair, continue, or separately consider expansion | | Reporting boundary | Claims the resulting evidence cannot support |
Worked example: contract-risk summary pilot
A synthetic in-house legal team is considering how to check capability after AI training. The decision is whether to add a practice exercise for staff who prepare first-pass contract-risk summaries. The legal leader reviews professional and confidentiality issues; legal operations owns the workflow; learning owns practice design; technology and security own the approved environment.
The team compares four options: current supervisor review, a knowledge quiz, manager observation, and a synthetic simulation. It selects a 20-person simulation pilot because the decision requires comparable evidence about source hierarchy, confidentiality, and verification. The scenario contains an executed agreement, a later draft amendment, a supplier email that conflicts with both, and a confidential note that must remain outside the AI tool.
The pilot reveals that several participants identify the commercial conflict but do not distinguish executed from draft text. Two reviewer disagreements expose an ambiguous rubric anchor. One participant has incomplete viewer telemetry, so no process inference is made for that record. The team chooses repair and rerun: clarify the source-status cue, add a reviewer calibration example, and retain a manual evidence path.
The result supports improving the bounded exercise. It does not establish organisation-wide capability, legal compliance, learning transfer, financial benefit, or future matter performance.
Two Prune boundary
Two Prune can provide realistic synthetic work, capture available process and deliverable evidence, and report observed assessment performance with evidence-strength and calibration notes. It does not provide legal advice, certify a legal professional, decide whether a legal workflow is compliant, replace authorised supervision, establish return on investment, or make employment decisions.
Primary sources
Solicitors Regulation Authority, Misuse of AI warning notice
- Solicitors Regulation Authority, Effective supervision
- Solicitors Regulation Authority, Risk Outlook: the use of artificial intelligence in the legal market
Sources: [1] [2] [3]
Sources
- 1.Solicitors Regulation Authority, Misuse of AI warning notice · www.sra.org.uk
- 2.Solicitors Regulation Authority, Effective supervision · www.sra.org.uk
- 3.Solicitors Regulation Authority, Risk Outlook: the use of artificial intelligence in the legal market · www.sra.org.uk