Practical guide
AI capability measurement guide for the enterprise risk leader
An enterprise risk leader needs an AI capability agenda that connects observed work practices to organizational objectives, risk appetite, controls, and escalation. The role is rarely the sole owner of adoption, learning, technology, privacy, or business performance. Its distinctive contribution is to make uncertainty and exposure decision-ready while keeping control owners, service owners, and.
An enterprise risk leader needs an AI capability agenda that connects observed work practices to organizational objectives, risk appetite, controls, and escalation. The role is rarely the sole owner of adoption, learning, technology, privacy, or business performance. Its distinctive contribution is to make uncertainty and exposure decision-ready while keeping control owners, service owners, and assurance functions in their proper roles.
Mandate and evidence boundary
HM Treasury’s Orange Book treats risk management as part of governance, leadership, options appraisal, monitoring, and useful reporting. It calls for clear responsibilities, best available information, constructive challenge, and integration with decision-making. The NIST AI Risk Management Framework organizes voluntary AI risk work across Govern, Map, Measure, and Manage, while its Playbook is a menu of suggested actions rather than a universal checklist. These sources support a risk agenda; they do not assign every responsibility to an enterprise risk leader or prove that a particular assessment programme produces business outcomes. HM Treasury: The Orange Book NIST: AI Risk Management Framework
Sources: [1] [2]
Build the agenda with RANGE
### R — Relate capability to objectives and decisions
Start with the objectives and decisions that AI-enabled work affects. Avoid beginning with a generic list of AI skills. Map each workflow to its intended outcome, material dependencies, affected parties, and plausible failure modes.
For each agenda item, state the decision required: permit a pilot, restrict a use, improve a control, fund learning, investigate an incident pattern, or request stronger evidence. Name the accountable decision forum and timing.
The enterprise risk leader should ask whether the organization can see the relevant uncertainty early enough to act. That is different from owning the workflow or deciding every acceptable use.
### A — Assign authority and lines of responsibility
Classify each stakeholder as owner, operator, reviewer, assurance provider, or collaborator. A service owner may own workflow outcomes; technology may own platform configuration; security and privacy functions own their professional controls; learning may own support; enterprise risk may maintain the integrated risk view and escalation route.
Record who can accept residual risk and at what level. Separate the person producing evidence from the person reviewing it. A role title alone is not evidence of budget, control, or decision authority.
### N — Name evidence layers and gaps
Use at least four evidence layers:
Do not substitute access, training completion, tool activity, assessment performance, and workplace outcomes for one another. Reconcile denominators and missingness. Keep evidence about observed performance separate from claims about operational impact.
### G — Gauge options against appetite and reversibility
For each material risk, compare options such as continue, restrict, repair, pause, replace, or gather evidence. Show expected benefit, downside, uncertainty, control burden, and reversibility. Relate the choice to the organization’s stated risk appetite without turning a broad appetite statement into automatic approval.
Define trigger conditions. Examples include a high-consequence error, uncontrolled sensitive-data transfer, repeated source fabrication, a failed access control, or unexplained performance drift. Triggers need owners, response times, and escalation destinations.
### E — Escalate, evaluate, and learn
Create a reporting rhythm proportionate to exposure. A new high-consequence workflow may need frequent review; a stable low-consequence use may need periodic sampling. Reports should surface movement, exceptions, evidence gaps, and the next decision rather than a single composite status.
After each pilot, incident, or material change, record what was learned, what control changed, and which assumption remains unresolved. Update the agenda when tools, data, processes, users, or external requirements change.
- Exposure: which workflows, information classes, users, and decisions are in scope.
- Practice: what people actually did, including source use, AI interaction, checking, escalation, and revision.
- Control operation: whether required approvals, restrictions, logs, reviews, and incident routes operated.
- Outcome: what changed in quality, time, cost, user experience, or risk, with a credible comparison where one exists.
Decision and evidence map
| Decision | Primary owner | Enterprise risk contribution | Minimum evidence | Escalation condition | | --- | --- | --- | --- | --- | | Permit a bounded pilot | service owner | integrated risk view and forum route | workflow, users, data, baseline, controls, stop triggers | material control absent | | Expand use | business executive | challenge evidence strength and concentration | representative practice evidence, control tests, outcome comparison | uncertainty exceeds appetite | | Restrict a workflow | control or service owner | aggregate exposure and coordinate escalation | incident pattern, affected population, alternative process | high-consequence event | | Fund capability support | learning or business owner | connect weaknesses to material risks | task-specific evidence, error patterns, intervention design | support misses priority exposure | | Accept residual risk | delegated authority | document basis and dependencies | options, controls, assurance, dissent, review date | authority or evidence unclear |
Worked agenda: synthetic claims-operations pilot
A financial-services organization is considering AI assistance for summarizing commercial claim files before human review. The enterprise risk leader is asked whether the pilot can expand from 20 specialists to the full unit.
The RANGE map begins with the decision: the claims service owner will choose among expansion, a restricted second cohort, or pause. Enterprise risk prepares the integrated view; information security, privacy, legal, model risk, quality assurance, and operations retain their professional or operational responsibilities.
Exposure evidence identifies three claim types, two information classifications, four source systems, and one downstream payment recommendation. Practice evidence covers 18 completed simulations rather than all 20 enrolled specialists. Two did not finish and remain missing, not low performers.
The work evidence shows strong document retrieval but recurring failures to distinguish policy language from adjuster notes. Three submissions omitted a contradictory attachment, and two accepted an unsupported AI summary. A separate control test finds that source links persisted in the workspace, but one export path removed them. No payment decision was made in the pilot.
The outcome layer is too weak for a unit-wide productivity claim: timing data covers only 11 comparable cases, case complexity was not balanced, and quality review used a new rubric. The agenda therefore presents three options. Immediate expansion offers reach but carries unresolved evidence and export-lineage risk. A pause allows redesign but discards a usable bounded workflow. A restricted second cohort repairs source persistence, stratifies cases, and tests reviewer agreement before another scale decision.
The recommended agenda item is the restricted cohort. Conditions are: no downstream payment automation; manual confirmation of cited policy and attachments; repaired export lineage; balanced claim categories; preregistered quality and timing measures; recorded non-completion; and a stop trigger for any unsupported recommendation reaching a live claim file.
The evidence artifact contains the workflow map, authority matrix, exposure register, participant funnel, observed-practice findings, control-test results, option paper, residual-risk record, and review date. The enterprise risk leader records the unresolved question—whether observed simulation behavior transfers to live operations—rather than treating it as answered.
Questions for the risk forum
Two Prune provides realistic AI-enabled work assessments and evidence about observed participant performance. It can contribute task-specific evidence to a wider risk picture. It does not certify control effectiveness, determine risk appetite, establish operational impact, or make employment decisions.
- Which business objective and decision does this evidence inform?
- What population and workflow are represented, and what is missing?
- Are practice, control operation, and outcome evidence being conflated?
- Which risks are concentrated across several workflows or providers?
- Who owns remediation, residual-risk acceptance, and independent review?
- What would trigger restriction, escalation, or a new assessment?
- What comparison supports any claimed operational change?
- What decision should the forum make now, and what remains for later?
Sources
HM Treasury: The Orange Book
- NIST: AI Risk Management Framework
- NIST AI RMF Playbook
Sources: [1] [2] [3]
Sources
- 1.HM Treasury: The Orange Book · UK Government
- 2.NIST: AI Risk Management Framework · National Institute of Standards and Technology
- 3.NIST AI RMF Playbook · National Institute of Standards and Technology