Practical guide
How to report AI capability evidence to a legal leader
A legal leader needs a decision packet, not a stream of activity metrics. Good reporting explains the workflow tested, the evidence observed, the material failures and unknowns, who reviewed them, and the next decision. It also distinguishes professional accountability from ownership of budget, technology, privacy, or workforce decisions. Use the CLEAR reporting method: Context and decision,.
A legal leader needs a decision packet, not a stream of activity metrics. Good reporting explains the workflow tested, the evidence observed, the material failures and unknowns, who reviewed them, and the next decision. It also distinguishes professional accountability from ownership of budget, technology, privacy, or workforce decisions. Use the CLEAR reporting method: Context and decision, Limits and coverage, Evidence and exceptions, Accountability, and Response. The result should let a legal leader challenge the evidence without mistaking a pilot result for approval of a tool or a person.
C — State the context and decision
Open with one decision and its date. Examples include extending a controlled workflow, repairing a supervision step, narrowing permitted use, commissioning more evidence, or stopping. Name the legal leader's role in that decision: owner, sponsor, reviewer, adviser, or informed stakeholder. Do not infer authority from title alone.
Define the workflow, users, materials, tool configuration, jurisdiction, observation window, comparison, and excluded uses. Report why the issue reaches legal leadership: professional duty, confidentiality, privilege, client impact, regulatory exposure, supervision, material contractual consequence, or another explicit reason.
The Solicitors Regulation Authority's warning notice on misuse of AI says existing professional obligations continue to apply when AI is used and highlights accuracy, confidentiality, human oversight, and proportionate risk management. For an SRA-regulated context, those duties should shape the report. The notice is not a universal legal mandate and does not determine the correct decision in another jurisdiction. SRA misuse of AI warning notice
Sources: [1]
L — Show limits, coverage, and the comparison
Report what the evidence can and cannot represent. Include the number of eligible cases, observed cases, reviewed artifacts, excluded records, missing events, withdrawn participants, reviewer coverage, and any shift in task or tool configuration. State whether tasks were synthetic, anonymised, or live and approved.
Keep comparison options visible. A report should not compare the proposed AI-assisted workflow only with an imaginary process that has no cost or error. Show the current process, a non-AI process improvement, a narrower controlled use, and stopping where relevant. Use the same definitions across options.
Avoid rolling access, usage, quality, control operation, capability, and business outcome into one score. A high usage rate may coexist with weak source checking; a strong controlled assessment may say little about live-work frequency. If the evidence is too sparse for the decision, label the decision hold for evidence rather than filling the gap with narrative.
E — Present evidence, exceptions, and disagreement
Organize the evidence around the decision. A useful table has columns for claim, observation, source, denominator, reviewer, exception, confidence or limitation, and decision relevance. Lead with critical errors, incidents, missing evidence, and unresolved reviewer disagreement before averages or success stories.
The SRA's effective-supervision guidance says AI-assisted or AI-generated work needs appropriate human review, scrutiny, and professional judgment, with an authorised individual retaining responsibility for legal services. Report the operation of that review: what was checked, by whom, against which standard, what changed, what escaped, and how disagreements were resolved. SRA effective supervision guidance
Preserve source anchors and versions. For every material finding, the legal leader should be able to inspect the original artifact, governing material, reviewer note, and disposition. A generated summary can orient the reader, but it cannot replace the underlying evidence.
Sources: [2]
A — Make accountability visible
Separate at least five responsibilities: legal accountability for the service or advice, workflow ownership, technology ownership, information governance, and the decision owner for the next stage. Add assessment or evaluation ownership when capability evidence comes from a simulation. One person may hold several roles, but the report should not collapse them.
Include a control register with the control, owner, trigger, observable evidence, exception path, and stop condition. Report whether each control was designed, tested, operated, failed, or was not observed. “Policy exists” and “no incident was reported” are not evidence that the control operated.
The SRA Risk Outlook on AI in legal services describes current and potential risks and opportunities for firms and consumers. Use it to prompt a context-specific risk review, not as a checklist that establishes acceptability. SRA Risk Outlook: The use of artificial intelligence in the legal market
Sources: [3]
R — Recommend a bounded response
End with a disposition: extend, repair and retest, narrow, hold, or stop. Link it to pre-agreed decision rules and name the owner and deadline for every follow-up. Separate immediate containment from longer-term improvement.
A bounded recommendation might say: “Repair the citation-verification checkpoint and repeat the calibration set before any broader use.” It should not say “AI is approved” when only one workflow and configuration were observed. If legal leadership is advisory rather than the final owner, record the legal view and route the operational decision to the accountable forum.
List the next evidence gate. Examples include successful recalibration, complete source linkage, independent review of high-severity cases, privacy approval, a larger representative sample, stable instrumentation, or evidence that review effort remains operationally viable.
Worked report: contract-risk summary pilot
This synthetic report concerns a controlled exercise in which a legal operations team used an approved AI tool to prepare first-pass contract-risk summaries from synthetic agreements. The legal leader is the professional-risk reviewer; the general counsel and operations director jointly own the extension decision.
Decision: whether to move from a 20-artifact exercise to a limited live pilot using approved low-risk contracts.
| Evidence area | Observation | Limitation | Decision relevance | | --- | --- | --- | --- | | Coverage | 20 of 20 synthetic artifacts submitted; two reviewers independently checked all critical categories | Synthetic documents do not reproduce every live matter condition | Adequate for calibration, not broad live approval | | Source traceability | 17 artifacts anchored every material issue to a clause; 3 had at least one weak anchor | No missing telemetry | Repair needed before live work | | Critical errors | One summary omitted a non-standard renewal term; human review caught it before disposition | Exercise cannot estimate a population error rate | Stop condition did not occur, but review remains essential | | Unsupported additions | Two artifacts introduced unsupported jurisdictional language; participants removed both after checking sources | The correction rule was interpreted differently by reviewers | Recalibration required | | Reviewer agreement | Reviewers matched on 16 artifacts and differed on four; two differences would change the control rating | Small calibrated set | Decision cannot rely on the average score alone | | Incidents | No prohibited data entered and no confidentiality incident observed | Absence in 20 synthetic cases is not evidence about all live use | Information-governance gate remains open |
The report recommends repair and retest. The team will rewrite the source-anchor requirement, add a second reviewer-calibration set, and repeat five challenge artifacts containing stale terms and conflicting clauses. A limited live pilot remains on hold until reviewer agreement is sufficient for the named decision, the privacy owner approves the data route, and the workflow owner demonstrates an auditable escalation path.
The appendix contains the protocol version, source pack hashes, all artifacts, reviewer scores before reconciliation, disagreement notes, correction histories, control register, missing-data log, and decision record. The report does not infer organisation-wide legal capability, endorse the tool for other matters, or convert exercise performance into an employment recommendation.
Two Prune boundary
Two Prune can run realistic work simulations and report evidence of observed participant performance across context framing, evidence navigation, AI orchestration, verification and risk control, judgment and synthesis, and communication. It is not a law firm, legal-services regulator, technology approval body, or automated employment-decision system. Customers define legal scope, approved materials and tools, supervision, privacy controls, decision rights, and operational use of the evidence.
Sources
- 1.SRA misuse of AI warning notice · www.sra.org.uk
- 2.SRA effective supervision guidance · www.sra.org.uk
- 3.SRA Risk Outlook: The use of artificial intelligence in the legal market · www.sra.org.uk