Practical guide
AI capability measurement agenda for a functional practice leader
A functional practice leader needs an agenda that turns AI capability evidence into a decision about standards, quality, review burden, and development. The meeting should not become a product demonstration or a debate about one average score. It should identify the work standard, compare evidence options, inspect failure patterns, and assign the next bounded test.
A functional practice leader needs an agenda that turns AI capability evidence into a decision about standards, quality, review burden, and development. The meeting should not become a product demonstration or a debate about one average score. It should identify the work standard, compare evidence options, inspect failure patterns, and assign the next bounded test.
Prepare the evidence pack
Send a short packet at least two working days before the meeting:
Do not send unrestricted participant logs when aggregated patterns and traceable exceptions answer the decision. Preserve appropriate access boundaries and give the leader a route to inspect supporting evidence when necessary.
- the decision and alternatives;
- the current work standard or quality criteria;
- the target workflow and representative task;
- available assessment, output, process, control, and resource evidence;
- exceptions, missingness, accessibility issues, and reviewer disagreement;
- current training, manager-review, analytics, or work-sample alternatives;
- authority map, constraints, and next decision date.
Run the seven-part PRACTICE agenda
### 1. Purpose: what decision must this meeting support? (5 minutes)
Choose one: define a capability standard, approve a bounded assessment, repair an existing approach, authorize a pilot, or stop work. Name the final decision owner and the date. Record what happens if evidence remains inconclusive.
### 2. Required work: what does competent performance look like? (10 minutes)
Identify a recurring, consequential workflow and its acceptable output. Define the sources, constraints, quality checks, escalation points, and human decisions involved. Separate the practice standard from preferred tool behavior. A useful standard can be demonstrated with or without one specific AI product.
### 3. Alternatives: what evidence is already available? (10 minutes)
Compare training completion, self-report, usage analytics, manager observation, final-output review, natural-work telemetry, and controlled simulations. For each option, state what it observes, what it misses, review cost, comparability, and data boundary. The meeting should improve an evidence portfolio, not force one instrument to answer every question.
### 4. Capability evidence: what was observed? (15 minutes)
Review distributions and failure patterns across task interpretation, evidence navigation, AI use, verification, judgment, and communication. Reconcile invited, started, completed, valid, excluded, and reviewed records. Put critical errors and missing evidence before averages. Do not convert a system-access failure into participant performance.
### 5. Tradeoffs and objections: what could make the evidence misleading? (10 minutes)
Test task realism, sample selection, source-set quality, tool configuration, accessibility, reviewer consistency, learning effects, and generalization. Ask whether the same decision could be supported more simply through process repair or manager calibration.
### 6. Intervention: what bounded next step follows? (10 minutes)
Choose scale, repair, hold, or stop. Assign owners for scenario design, reviewer calibration, accessibility, policy, data handling, and participant communication. Set resource limits and the decision that must remain human.
### 7. Check and expiry: what evidence closes the next gate? (5 minutes)
Define the next task, population, comparison, evidence fields, critical-failure rule, reviewer, and review date. Expire the conclusion when the workflow, standard, tools, source set, or participant context changes materially.
The UK government’s governance and management framework guide defines such a framework as authority limits, decision roles, autonomy, assurance needs, reporting, accountabilities, practices, and documentation. It also says an “established” way of working is implemented and used, not merely documented. This distinction helps a practice leader ask for operating evidence rather than policy text alone. Guide to governance and management frameworks
Sources: [2]
Copyable decision and evidence map
| Decision question | Current evidence option | What it observes | Material gap | Resource cost | Authority | Next gate | | --- | --- | --- | --- | --- | --- | --- | | define / pilot / repair / scale / stop | training, analytics, observation, output review, simulation | bounded signal | missing evidence or confound | participant and reviewer effort | owner / influencer / reviewer / collaborator | test, threshold, date |
Add a standard-to-evidence matrix:
| Practice expectation | Representative task opportunity | Observable evidence | Critical failure | Reviewer rule | Follow-up action | | --- | --- | --- | --- | --- | --- | | concrete expectation | task feature | output and process evidence | predefined condition | scoring and escalation rule | development, repair, or retest |
Worked example: proposal-quality practice
This synthetic example concerns a leader responsible for proposal quality in a professional-services practice. The practice has provided AI training and can see tool activity, but review teams still report unsupported claims and missed requirements. The meeting decision is whether to run a controlled assessment before changing the practice standard or buying more training.
The Required work discussion narrows the target to a proposal compliance and red-team workflow. Competent performance includes mapping requirements, locating approved evidence, identifying unsupported assertions, checking calculations, escalating ambiguity, and delivering a usable correction log. The practice leader owns the quality expectation but collaborates with the AI programme sponsor and security reviewer on the assessment boundary.
The Alternatives comparison shows that course completion records participation, usage analytics record activity, and final proposal review finds some defects after work is produced. A controlled simulation could create a consistent opportunity to observe requirement mapping and verification, but it will not reproduce every client, deadline, or team condition.
The Capability evidence available from a small earlier exercise is not comparable because participants received different source packs. The meeting does not average those scores. Instead, it records the useful failure taxonomy and treats the performance conclusion as unavailable.
The disposition is repair and pilot: build one synthetic source pack with a planted contradiction and unsupported claim, calibrate two reviewers, include an accessible task route, and run a small cohort. The next gate asks whether reviewers can trace findings consistently and whether the evidence changes a defined development decision. Expansion remains held until that gate is met.
Objections and reporting boundary
If the leader asks for a single ranking, return to the practice decision and the evidence limits. If they ask for business impact, separate task performance from operational quality, rework, client outcome, or revenue. Those larger claims need their own comparison and data. If they already have rich workplace evidence, a controlled assessment may add little and the correct decision may be to improve existing review practices.
Hold the meeting decision when the practice standard is undefined, authority is unclear, the workflow is not representative, evidence counts cannot be reconciled, critical exceptions are hidden, reviewer rules are absent, or the proposed action would proceed regardless of results.
Two Prune can provide realistic work simulations and evidence-backed assessment records for a bounded capability discussion. It does not prove practice-wide outcomes, replace the functional leader’s judgment, or make employment decisions.
Sources
UK Cabinet Office: Government Functional Standard GovS 001, Government functions
- UK Cabinet Office: Guide to governance and management frameworks
Sources: [1] [2]
Sources
- 1.GovS 001: Government functions · UK Government
- 2.Guide to governance and management frameworks · UK Government