Practical guide

AI capability measurement guide for a legal leader

A legal leader's AI capability agenda should focus on the decisions that expose clients, matters, the organisation, or the administration of justice to material risk. It should not turn legal into the owner of every tool or reduce capability to course completion. The legal leader is commonly an adviser, reviewer, control owner for defined issues, or collaborator with business, technology,.

By Two Prune

A legal leader's AI capability agenda should focus on the decisions that expose clients, matters, the organisation, or the administration of justice to material risk. It should not turn legal into the owner of every tool or reduce capability to course completion. The legal leader is commonly an adviser, reviewer, control owner for defined issues, or collaborator with business, technology, privacy, security, learning, and professional-supervision owners. Use CASE to build the agenda: classify the work and authority, align evidence to professional obligations, set assessment and supervision questions, and escalate exceptions through explicit gates. The output is a decision agenda and evidence map, not a universal scorecard for lawyers.

C: Classify the work, consequence, and decision authority

Start with the actual AI-assisted task. Record what information enters the system, what the tool produces, who relies on the output, whether the work reaches a client, court, regulator, counterparty, or internal decision-maker, and what happens if it is wrong. Separate administrative assistance from legal research, drafting, analysis, advice, filing, disclosure, or matter strategy.

Map authority before requesting measurement. The legal leader may own a professional-risk decision, advise on law and privilege, review provider terms, or set supervision expectations. The business owner still owns the workflow outcome. Technology and security owners manage configuration and access. Privacy and records owners retain their responsibilities. A regulated professional remains responsible for work that professional rules place on them.

The Solicitors Regulation Authority's 2026 warning notice identifies inaccurate AI-generated legal work and confidential information entered into public tools as observed concerns. Its supervision guidance says authorised individuals retain ultimate responsibility for legal services delivered with AI assistance. These are reasons to make task and consequence visible, not to assume all AI use is prohibited.

A: Align evidence to duties and operating controls

Create an evidence map for each material obligation. Depending on jurisdiction and practice, questions may include competence, confidentiality, privilege, client communication, candour, supervision, conflicts, records, fees, supplier terms, and court or regulator requirements. Use current local authority and qualified review rather than treating one professional body's guidance as globally applicable.

The American Bar Association's Formal Opinion 512 addresses competence, confidentiality, communication, candour, supervision, and fees for generative AI use. It is authoritative professional guidance within its stated context, not a substitute for applicable rules elsewhere.

For each issue, record:

Do not use prompt volume, tool activation, or self-reported confidence as a proxy for professional capability. Those measures may describe adoption or experience. They do not show whether a person protected information, checked authorities, found a fabricated citation, or exercised sound judgment.

  • the decision and accountable authority;
  • the applicable source and jurisdiction;
  • the task conditions that create the issue;
  • the expected behaviour or control;
  • evidence available from the deliverable and work process;
  • missingness and alternative explanations;
  • the threshold for pause, escalation, or specialist review.

S: Set the assessment and supervision questions

Choose realistic tasks that create observable evidence without using live client information. A synthetic assessment might ask a participant to review a draft legal brief containing an outdated authority, a fabricated citation, an ambiguous instruction, and confidential facts that should not be placed in an unapproved tool.

Define what evidence the task can and cannot support. Relevant observations may include whether the participant frames the matter, selects authoritative sources, limits sensitive inputs, instructs AI appropriately, verifies citations and quotations, identifies uncertainty, escalates a conflict, revises the draft, and communicates a bounded conclusion.

Keep final output and process evidence together. A polished memo can conceal unverified research. A correction log can show that a participant detected and repaired an error. Missing process events should reduce confidence rather than be interpreted as a failure by the participant.

Connect the assessment to supervision design. The SRA says AI-assisted or generated legal work requires appropriate human review, scrutiny, and professional judgment. The agenda should therefore ask which work requires review, who is authorised to review it, what evidence the reviewer needs, and what must be retained.

E: Escalate decisions through explicit evidence gates

End each agenda item with a decision route. Options may include prohibit the task for the proposed tool, allow a synthetic experiment, permit bounded internal use with controls, require specialist review, repair the process and rerun, or approve broader use through a separate governance decision.

Set change triggers. A new tool, model, provider term, data route, jurisdiction, matter type, external audience, automation level, or incident should reopen the relevant decision. Approval for summarising public guidance does not automatically extend to analysing privileged matter files.

Worked example: a synthetic regulatory-response exercise

A synthetic in-house legal team wants to decide whether a realistic assessment should be added after AI training. The proposed task asks participants to review a draft response to a regulator. The materials contain a current rule, a superseded guidance note, a fabricated case citation, an ambiguous business claim, and a note labelled privileged.

CASE identifies four evidence opportunities. Participants should keep the privileged note out of the unapproved assistant, verify the case and rule against authoritative sources, resolve or surface the conflicting guidance dates, and explain the unsupported business claim to the response owner. The legal leader reviews the professional-risk questions; L&D owns the practice design; the regulatory team defines acceptable work quality; privacy and security approve the synthetic tool configuration.

The pilot report shows 18 of 24 reviewable submissions catch the fabricated citation, while only nine explain why the superseded guidance changes the response. Three submissions lack complete viewer events and are excluded from process-pattern claims. The evidence supports adding a coached source-hierarchy exercise and rerunning the same bounded task. It does not establish general legal competence, future matter performance, or training impact.

Measurement and Two Prune boundary

Two Prune can provide a synthetic legal-work simulation, capture available workflow and deliverable evidence, and report observed assessment performance with evidence-strength and calibration notes. It does not provide legal advice, determine professional compliance, certify a lawyer, prove learning transfer, predict future performance, or make employment decisions. Any benchmark should remain directional or unavailable until supported by an appropriate comparison population and method.

Primary sources

Solicitors Regulation Authority, Misuse of AI warning notice

  • Solicitors Regulation Authority, Effective supervision
  • American Bar Association, Formal Opinion 512

Sources: [1] [2] [3]