Practical guide
AI capability measurement guide for a legal leader
A legal leader's AI capability agenda should focus on the decisions that expose clients, matters, the organisation, or the administration of justice to material risk. It should not turn legal into the owner of every tool or reduce capability to course completion. The legal leader is commonly an adviser, reviewer, control owner for defined issues, or collaborator with business, technology,.
A legal leader's AI capability agenda should focus on the decisions that expose clients, matters, the organisation, or the administration of justice to material risk. It should not turn legal into the owner of every tool or reduce capability to course completion. The legal leader is commonly an adviser, reviewer, control owner for defined issues, or collaborator with business, technology, privacy, security, learning, and professional-supervision owners. Use CASE to build the agenda: classify the work and authority, align evidence to professional obligations, set assessment and supervision questions, and escalate exceptions through explicit gates. The output is a decision agenda and evidence map, not a universal scorecard for lawyers.
A: Align evidence to duties and operating controls
Create an evidence map for each material obligation. Depending on jurisdiction and practice, questions may include competence, confidentiality, privilege, client communication, candour, supervision, conflicts, records, fees, supplier terms, and court or regulator requirements. Use current local authority and qualified review rather than treating one professional body's guidance as globally applicable.
The American Bar Association's Formal Opinion 512 addresses competence, confidentiality, communication, candour, supervision, and fees for generative AI use. It is authoritative professional guidance within its stated context, not a substitute for applicable rules elsewhere.
For each issue, record:
Do not use prompt volume, tool activation, or self-reported confidence as a proxy for professional capability. Those measures may describe adoption or experience. They do not show whether a person protected information, checked authorities, found a fabricated citation, or exercised sound judgment.
- the decision and accountable authority;
- the applicable source and jurisdiction;
- the task conditions that create the issue;
- the expected behaviour or control;
- evidence available from the deliverable and work process;
- missingness and alternative explanations;
- the threshold for pause, escalation, or specialist review.
S: Set the assessment and supervision questions
Choose realistic tasks that create observable evidence without using live client information. A synthetic assessment might ask a participant to review a draft legal brief containing an outdated authority, a fabricated citation, an ambiguous instruction, and confidential facts that should not be placed in an unapproved tool.
Define what evidence the task can and cannot support. Relevant observations may include whether the participant frames the matter, selects authoritative sources, limits sensitive inputs, instructs AI appropriately, verifies citations and quotations, identifies uncertainty, escalates a conflict, revises the draft, and communicates a bounded conclusion.
Keep final output and process evidence together. A polished memo can conceal unverified research. A correction log can show that a participant detected and repaired an error. Missing process events should reduce confidence rather than be interpreted as a failure by the participant.
Connect the assessment to supervision design. The SRA says AI-assisted or generated legal work requires appropriate human review, scrutiny, and professional judgment. The agenda should therefore ask which work requires review, who is authorised to review it, what evidence the reviewer needs, and what must be retained.
E: Escalate decisions through explicit evidence gates
End each agenda item with a decision route. Options may include prohibit the task for the proposed tool, allow a synthetic experiment, permit bounded internal use with controls, require specialist review, repair the process and rerun, or approve broader use through a separate governance decision.
Set change triggers. A new tool, model, provider term, data route, jurisdiction, matter type, external audience, automation level, or incident should reopen the relevant decision. Approval for summarising public guidance does not automatically extend to analysing privileged matter files.
Legal-leader decision agenda artifact
| Agenda item | Decision | Legal leader's role | Other owner | Evidence required | Stop or escalation trigger | |---|---|---|---|---|---| | Task boundary | Which AI-assisted legal tasks are in scope? | Adviser/reviewer | Practice owner | Task map and consequence analysis | External reliance or new matter type | | Information | Which data may enter which tool? | Issue owner or adviser | Privacy/security owner | Data flow, terms, configuration, test | Confidential or privileged data route unclear | | Verification | What review is required before release? | Reviewer/collaborator | Authorised supervisor | Source trace, citation checks, review record | Authority cannot be verified | | Capability evidence | What should realistic practice observe? | Collaborator | L&D/assessment owner | Synthetic task, rubric, evidence map | Live client data or unsupported inference | | Exceptions | Who can pause, approve, and close an exception? | Defined by issue | Workflow owner | Exception record and closure test | Repeated bypass or unresolved harm |
For each row, add the applicable jurisdiction, decision date, source version, owner, review date, and conditions. The table is a routing device; it is not legal advice.
Worked example: a synthetic regulatory-response exercise
A synthetic in-house legal team wants to decide whether a realistic assessment should be added after AI training. The proposed task asks participants to review a draft response to a regulator. The materials contain a current rule, a superseded guidance note, a fabricated case citation, an ambiguous business claim, and a note labelled privileged.
CASE identifies four evidence opportunities. Participants should keep the privileged note out of the unapproved assistant, verify the case and rule against authoritative sources, resolve or surface the conflicting guidance dates, and explain the unsupported business claim to the response owner. The legal leader reviews the professional-risk questions; L&D owns the practice design; the regulatory team defines acceptable work quality; privacy and security approve the synthetic tool configuration.
The pilot report shows 18 of 24 reviewable submissions catch the fabricated citation, while only nine explain why the superseded guidance changes the response. Three submissions lack complete viewer events and are excluded from process-pattern claims. The evidence supports adding a coached source-hierarchy exercise and rerunning the same bounded task. It does not establish general legal competence, future matter performance, or training impact.
Measurement and Two Prune boundary
Two Prune can provide a synthetic legal-work simulation, capture available workflow and deliverable evidence, and report observed assessment performance with evidence-strength and calibration notes. It does not provide legal advice, determine professional compliance, certify a lawyer, prove learning transfer, predict future performance, or make employment decisions. Any benchmark should remain directional or unavailable until supported by an appropriate comparison population and method.
Primary sources
Solicitors Regulation Authority, Misuse of AI warning notice
- Solicitors Regulation Authority, Effective supervision
- American Bar Association, Formal Opinion 512
Sources: [1] [2] [3]
Sources
- 1.Solicitors Regulation Authority, Misuse of AI warning notice · www.sra.org.uk
- 2.Solicitors Regulation Authority, Effective supervision · www.sra.org.uk
- 3.American Bar Association, Formal Opinion 512 · www.americanbar.org