Practical guide
AI adoption and quality scorecard template
Use this scorecard to report whether an approved AI-enabled workflow is being used, whether reviewed work meets its quality standard, whether controls operate, and what decision the evidence supports. It deliberately keeps adoption, quality, risk, capability, and outcome separate. A high activity rate cannot compensate for weak work or a critical control failure.
Use this scorecard to report whether an approved AI-enabled workflow is being used, whether reviewed work meets its quality standard, whether controls operate, and what decision the evidence supports. It deliberately keeps adoption, quality, risk, capability, and outcome separate. A high activity rate cannot compensate for weak work or a critical control failure.
Source basis and limits
The UK Government's AI impact evaluation framework treats adoption and real-world performance as a distinct evaluation category and recommends adapting metrics to the deployment context and available data. NIST's AI RMF Measure function calls for quantitative, qualitative, or mixed methods, documented test conditions, uncertainty, representative evaluation, regular monitoring, and independent review where appropriate. GOV.UK performance guidance says metrics should follow a service purpose and hypothesis, use defined sources, carry context, and inform improvement. These sources support the scorecard's design; they do not validate one composite score or a universal threshold. UK Government: AI impact evaluation framework NIST AI RMF Core—Measure GOV.UK: Set performance metrics
Sources: [1] [2] [3]
Programme header
Complete this before adding results.
| Field | Entry | | --- | --- | | decision and decision owner | | | named workflow and user | | | approved tool, model, plan, permissions, and configuration | | | eligible population | | | reporting window | | | baseline or comparison | | | quality standard and reviewer | | | prohibited failures and stop rules | | | privacy, access, retention, and consultation boundary | | | metric version and source snapshot | | | report owner and next review | |
Layered scorecard
Do not fill a row until its metric contract is complete.
| Layer | Decision question | Numerator | Denominator | Window | Source | Result | Threshold or comparison | Missingness and uncertainty | Owner | Disposition | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | eligibility and access | Can the intended population use the approved setup? | | | | | | | | | | | activation | Who completed the defined first useful action? | | | | | | | | | | | repeated use | Who used the named workflow at the required cadence? | | | | | | | | | | | workflow integration | What share of eligible target tasks used the approved process? | | | | | | | | | | | artifact quality | What share of reviewed outputs met the release standard? | | | | | | | | | | | control operation | Were required checks, escalations, and approvals performed? | | | | | | | | | | | applied capability | What behavior appeared in a controlled representative task? | | | | | | | | | | | operational outcome | Did the bounded workflow measure change under an appropriate design? | | | | | | | | | |
Metric contract
Create a contract for every populated row:
| Field | Definition | | --- | --- | | metric ID and version | stable identifier and effective date | | construct | the thing the metric is intended to represent | | decision | action this evidence can inform | | numerator | qualifying events or units and exclusions | | denominator | eligible population, tasks, outputs, or opportunities | | source and fields | system, query, review artifact, survey, or assessment evidence | | collection and refresh | when and how data is captured | | quality checks | completeness, accuracy, duplication, applicability, and reconciliation | | segmentation | workflow, case type, tool version, access condition, or risk tier | | missingness | what absent data means and how it is reported | | threshold | trigger for inquiry, action, pause, or escalation | | interpretation limit | what the metric cannot establish | | owner and reviewer | accountable maintainer and independent challenger where needed |
For proportions, always show the count and denominator beside the percentage. For sampled artifact review, show the sampling frame, selection method, case mix, reviewer method, and number excluded. Treat unavailable evidence as unavailable, not zero.
Quality and control review card
Activity telemetry rarely shows whether outputs were fit for purpose. Pair it with a reviewed artifact sample.
| Artifact ID | Case type | Tool and workflow version | Material defects | Unsupported claims | Source conflicts handled | Required review completed | Escalation correct | Release disposition | Reviewer | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | | | | | | | | | |
Predefine critical failures that cannot be offset by a total, such as release without required human approval, exposure of restricted data, fabricated evidence, inaccessible required interaction, or suppression of a material conflict. Record reviewer disagreement and resolution rather than hiding it in the average.
Interpretation and decision note
Complete this after the rows are reviewed:
Do not calculate an overall percentage unless a real decision requires it and the weighting, compensation rules, and critical failures are explicitly justified. Even then, retain every layer and denominator.
- Observed: exact result, denominator, period, and source.
- Compared with: baseline, matched group, threshold, or no defensible comparison.
- Possible explanations: at least two mechanisms, including instrumentation or selection effects.
- Contrary evidence: observation that weakens the preferred explanation.
- Missing: cases, fields, users, channels, or periods not observed.
- Decision: continue, investigate, repair, narrow, pause, or extend.
- Next evidence: named owner, date, method, and condition that would change the decision.
Worked scorecard: synthetic customer-response workflow
A fictional customer-operations team tests an approved assistant that drafts responses from a controlled knowledge base. The scorecard covers a four-week window. Eighty staff are eligible; 72 have working access after eight delayed permissions are repaired. Fifty-four complete a defined first useful action. Forty use the assistant in the named response workflow at least weekly.
Task telemetry contains 1,120 assisted drafts, but the organization cannot reliably count all eligible responses because one legacy channel is not instrumented. Workflow integration is therefore reported as unavailable rather than 1,120 / unknown. Repeated use is 40 / 72 accessible users, with the access denominator shown separately from the intended 80-person population.
Reviewers select 48 outputs using a preset sample: 24 ordinary queries, 12 policy exceptions, eight conflicting-source cases, and four accessibility-related cases. Thirty-nine meet the release standard on first review. Six require material correction, two omit a required escalation, and one contains a policy date unsupported by the cited article. The quality result is 39 / 48 for the sampled case mix, not 81% of all work.
The control card shows both missed escalations occurred on exception cases. The unsupported date appears after a knowledge-base migration and prompts a check of source freshness. Reviewer agreement is 45 of 48 initial dispositions; three are reconciled using the rubric. No restricted-data event is observed, but the report states that logging alone cannot establish absence across the uninstrumented channel.
An assessment of 16 volunteers on a separate, controlled case finds that 11 preserve a supplied source conflict and five select the more recent-looking statement without escalation. This is reported as task-bounded capability evidence with volunteer-selection limits, not added to the production quality numerator.
The decision is narrow and repair. The workflow continues for ordinary cases, exception cases require an added review gate, the knowledge-base migration is investigated, and access remediation remains open for eight staff. A later scorecard will test whether the two repairs change exception defects. The evidence does not establish productivity, organization-wide capability, or customer impact.
Scorecard QA
[ ] Every row answers a named decision and uses a versioned metric contract.
- [ ] Eligibility, access, activation, repeated use, and workflow integration remain distinct.
- [ ] Adoption is not used as a proxy for quality, capability, or outcome.
- [ ] Counts, denominators, periods, sources, exclusions, and missingness are visible.
- [ ] Artifact samples identify the frame, method, case mix, and reviewer process.
- [ ] Critical failures remain non-compensatory.
- [ ] Tool, model, plan, permissions, configuration, and workflow versions are recorded.
- [ ] Instrumentation changes and release events are annotated.
- [ ] Quantitative results are interpreted with qualitative and contextual evidence.
- [ ] Privacy, accessibility, retention, access, and consultation boundaries are reviewed.
- [ ] Comparisons and thresholds have an explicit rationale.
- [ ] Claims do not exceed the evidence design.
Adaptation guidance
For a training programme, add practice completion and delayed transfer as separate rows; do not treat attendance as transfer. For a high-risk workflow, expand control operation and incident review before adding outcome measures. For a small team, use counts and case narratives instead of unstable percentages. When telemetry coverage changes, version the metric and show a series break or defensible reprocessing rather than silently joining incomparable periods.
Version history
| Version | Effective date | Metric or field changed | Reason | Affected periods | Approved by | | --- | --- | --- | --- | --- | --- | | 0.1 | | initial decision design | | | | | 1.0 | | approved definitions and sources | | | | | | | | | | |
Two Prune boundary
Two Prune can contribute controlled assessment evidence about how participants frame a task, navigate sources, use AI, verify outputs, exercise judgment, and communicate a deliverable. That evidence can occupy the applied-capability layer when the task and interpretation align. Two Prune does not supply all adoption telemetry, prove workplace transfer or operational outcomes, validate a universal scorecard, monitor employees, or make employment decisions.
Sources
UK Government: AI impact evaluation framework
- NIST: AI Risk Management Framework Core
- GOV.UK Service Manual: How to set performance metrics for your service
Sources: [1] [2] [3]
Sources
- 1.UK Government: AI impact evaluation framework · architecture.cddo.cabinetoffice.gov.uk
- 2.NIST AI RMF Core—Measure · National Institute of Standards and Technology
- 3.GOV.UK: Set performance metrics · UK Government