Practical guide

AI metric definition verification checklist

Use this checklist after a metric definition has been drafted and before it is implemented in a dashboard, report, target, or decision rule. It tests whether another analyst can reconstruct the intended population and calculation, whether edge cases produce the expected result, and whether a reader can interpret movement without inventing meaning. A clean formula is not enough when the event,.

By Two Prune

Use this checklist after a metric definition has been drafted and before it is implemented in a dashboard, report, target, or decision rule. It tests whether another analyst can reconstruct the intended population and calculation, whether edge cases produce the expected result, and whether a reader can interpret movement without inventing meaning. A clean formula is not enough when the event, denominator, source boundary, latency, or permitted use remains ambiguous.

What this checklist can verify

The Government Digital Service distinguishes raw measurements from combined metrics and shows that a rate depends on correctly defined attempts and completions. The Department for Business, Innovation, Science and Trade's 2026 digital-indicator methodology warns that transaction definitions differ by service and that analytics collection has coverage limitations. The Department for Environment, Food and Rural Affairs' 2026 quality statement defines statistical quality as fitness for purpose and points to methodology, presentation, and quality-assurance tools. These current primary sources support a reproducibility review; they do not validate a particular organisation's metric. GDS: Using performance data to improve your service BIST digital-service indicators methodology Defra statistics quality statement

Sources: [1] [2] [3]

Run the RECON review

Mark each item pass, fail, not applicable, or not evidenced. Every pass needs a definition field, retained source, reproducible query, or test result. Do not average a critical failure into an overall percentage.

### R: Reconstruct the decision and construct

Stop condition: return the definition if the same number could trigger materially different actions for different readers, or if the label claims an outcome while the calculation measures only activity.

### E: Examine the eligible population

Stop condition: hold implementation when numerator and denominator use different grains, the eligible population cannot be reproduced, or a partial observed population is described as complete.

### C: Calculate independently

Stop condition: do not release a metric whose sampled result cannot be independently reproduced or whose missing values are silently converted into success, failure, or zero.

### O: Observe boundary and behaviour tests

Stop condition: return the definition when a boundary case reverses classification without an explicit rule, or when a team can improve the number by worsening the underlying service.

### N: Note interpretation and release controls

Stop condition: do not present a trend as comparable across a material definition change unless the historical series was legitimately restated and documented.

  • [ ] The metric names the decision, user, cadence, alternatives, and material error it supports.
  • [ ] The construct is classified as input, activity, output, quality, outcome, risk, or balancing measure.
  • [ ] The plain-language question matches the formula and dashboard label.
  • [ ] Permitted and prohibited uses are explicit.
  • [ ] Related measures do not silently stand in for the construct, such as activity for quality or attendance for transfer.
  • [ ] The unit is stable: person, account, case, session, transaction, event, or another named entity.
  • [ ] The observation window, time zone, event time, source cut-off, and maturity delay are stated.
  • [ ] Numerator and denominator each have separate inclusion and exclusion rules.
  • [ ] Duplicate, merged, retried, reopened, cancelled, test, deleted, late, and missing records have documented treatment.
  • [ ] Tracking consent, browser controls, offline activity, sampling, or source-system exclusions are visible.
  • [ ] The grouping key prevents one entity from entering the population more than intended.
  • [ ] Every field maps to an authoritative source and retains extraction time, definition, and owner.
  • [ ] Filters, joins, aggregations, thresholds, units, signs, null rules, and rounding are executable.
  • [ ] A second reviewer reproduces sampled values from source records.
  • [ ] The denominator-zero rule and minimum-denominator presentation rule are defined.
  • [ ] Reconciliation ties the metric population to a suitable source-system total.
  • [ ] AI-drafted SQL or field mappings have been checked against source semantics.
  • [ ] Tests cover just-inside and just-outside timestamps, thresholds, and eligibility rules.
  • [ ] Duplicate and reordered events do not change the result unexpectedly.
  • [ ] Late events and restatements follow the declared policy.
  • [ ] Changing one input affects only the intended metric and dependent views.
  • [ ] Synthetic positive, negative, ambiguous, and incomplete cases produce the expected classification.
  • [ ] A balancing measure or anti-gaming check covers the most likely distortion.
  • [ ] Direction of improvement, unit, precision, comparison basis, and required context are defined.
  • [ ] The definition distinguishes trend, target, benchmark, and causal explanation.
  • [ ] Limitations and uncertainty are visible beside the metric, not hidden in a distant document.
  • [ ] The owner, reviewer, version, effective date, approval, and next review date are recorded.
  • [ ] Changes to source, population, formula, workflow, AI contribution, purpose, or consequence trigger review.
  • [ ] Comparability breaks and backfill decisions are disclosed.

Verification record

| Area | Result | Evidence IDs | Failure or uncertainty | Owner | Release consequence | | --- | --- | --- | --- | --- | --- | | decision and construct | | | | | | | population | | | | | | | calculation and lineage | | | | | | | boundary and behaviour tests | | | | | | | interpretation and release | | | | | |

Choose one disposition: verified for the stated use, verified with visible limitations, return for repair, or hold implementation. The disposition applies only to the reviewed version and intended use.

Worked verification: synthetic first-pass resolution metric

A fictional service team defines first-pass resolution as cases closed after one substantive human response with no transfer and no reopen event during the next seven complete days. The metric will trigger monthly workflow investigation, not individual performance evaluation.

The test fixture contains 16 synthetic cases: a normal resolution, a transfer, a duplicate merge, spam, a test record, a case awaiting customer information, a reopen on day six, a reopen on day eight, two end-of-month cases that are not mature, a daylight-saving boundary, a blank closure time, and four ordinary unresolved cases.

RECON finds that the written population excludes spam and tests but the draft query excludes neither. It also uses message ID rather than case ID, so the merged duplicate appears twice. The day-six reopen incorrectly counts as resolved because the query compares calendar dates rather than elapsed time. The end-of-month cases are included even though the seven-day look-forward is incomplete.

After repair, 10 cases are eligible and mature. Six meet the first-pass rule, producing 60.0%. The original query reported 76.9%. The reviewer reconciles the 10 eligible cases to the retained source extract, confirms the day-eight reopen remains outside the declared window, and tests a zero-denominator month. The dashboard now labels the current month provisional until all cases mature and shows reopen rate as a balancing measure.

The disposition is verified with visible limitations. The metric can support investigation of this workflow after the maturity delay. It cannot establish service quality by itself, compare unrelated services with different case definitions, or explain why the rate changed.

Two Prune boundary

Two Prune can assess whether a participant verifies a supplied metric definition, reconstructs its population and calculation, challenges edge cases, and communicates a bounded disposition. It reports observed assessment evidence. It does not certify analytics systems, approve organisational targets, predict workplace outcomes, or make employment decisions.

Sources

Government Digital Service: Using performance data to improve your service

  • Department for Business, Innovation, Science and Trade: Digital-service indicators methodology
  • Department for Environment, Food and Rural Affairs: Statistics quality statement

Sources: [1] [2] [3]