Practical guide

How to create a metric definition with AI

A useful metric definition is a controlled specification, not a short label beside a dashboard tile. It states the decision the metric supports, the event and population being counted, the formula, exclusions, data lineage, quality rules, interpretation limits, owner, and change history. AI can help surface ambiguous terms and generate edge cases, but it should not choose a denominator, repair.

By Two Prune

A useful metric definition is a controlled specification, not a short label beside a dashboard tile. It states the decision the metric supports, the event and population being counted, the formula, exclusions, data lineage, quality rules, interpretation limits, owner, and change history. AI can help surface ambiguous terms and generate edge cases, but it should not choose a denominator, repair missing data, or change the business meaning without accountable review.

Build the definition with SPEC

Use four connected parts: Stake the decision, Pin the population, Express the calculation, Control interpretation and change. The output is a metric definition record that another analyst can implement and a decision maker can interpret without relying on undocumented team knowledge.

Current UK government guidance illustrates why this matters. The Government Digital Service distinguishes measurements from metrics and shows that a completion rate combines completed tasks with attempts. The Department for Business, Innovation, Science and Trade's current digital-service indicator methodology warns that transaction definitions differ by service and that web-analytics limitations affect interpretation. The Department for Energy Security and Net Zero monitoring framework connects indicators to intended use, decision makers, data quality, and whether a measure tracks inputs, outputs, outcomes, or objectives. These are primary sources for disciplined design; they do not prescribe a universal metric for every organisation. GDS: Using performance data to improve your service BIST digital-service indicators methodology DESNZ monitoring and evaluation framework

### S: Stake the decision

Write the decision before naming the metric. State who uses it, when, what alternative actions exist, and what error would matter. “Improve support quality” is not enough. “Decide each month whether to investigate a workflow, revise guidance, or leave it unchanged” gives the metric a bounded use.

Classify the construct: input, activity, output, service quality, outcome, risk, or balancing measure. Then write what the metric does not establish. A response-time metric does not establish resolution quality. Training attendance does not establish transfer. AI-tool activity does not establish sound judgment. A task-bounded assessment score does not represent a universal personal trait.

The definition record should include a decision ID, metric name, plain-language question, accountable decision owner, review cadence, materiality threshold, related measures, and prohibited uses.

### P: Pin the event and population

Define the unit of analysis and eligibility rules. Name the entity, event, time zone, observation window, source cut-off, inclusion rule, exclusion rule, and handling of duplicates, retries, reopened cases, deleted records, late events, and missing consent or tracking.

For a rate, define numerator and denominator as separate populations before writing the formula. A denominator is not “all users” unless the source can identify all eligible users. If browser controls, cookie consent, ad blockers, or offline activity remove observations, state that limitation rather than treating observed analytics as a complete population.

Preserve grain. A person, account, case, session, transaction, and event are not interchangeable. If the numerator counts cases and the denominator counts sessions, the resulting percentage may be calculable but meaningless. Record grouping keys and the rule for assigning one event to a period.

### E: Express the calculation and lineage

Write the formula in words and executable notation. Define every field, unit, filter, join, aggregation, threshold, rounding rule, and null treatment. Link each field to an authoritative source and record extraction time, transformation, and owner.

For example:

text eligible cases = closed support cases with a valid channel and closure timestamp in the month first-pass resolved cases = eligible cases with no reopen event within 7 complete days first-pass resolution rate = first-pass resolved cases / eligible cases

Add data-quality tests: unique-key checks, valid values, timestamp order, denominator nonzero, reconciliation to the source system, and boundary cases. Define latency and restatement. If a seven-day look-forward is required, the current month cannot be final on its last day.

AI can propose alternative wording, identify synonyms in source fields, draft SQL, and generate edge cases. Require a person to verify the source semantics and approve the executable definition. Never allow an AI tool to fill missing values or reinterpret statuses silently.

### C: Control interpretation and change

Set presentation rules: unit, decimal precision, direction of improvement, reporting interval, uncertainty or suppression rule, minimum denominator, comparison basis, and required context. Avoid targets until the team understands baseline quality and behavioural incentives.

Add balancing measures and anti-gaming checks. A faster-resolution metric may need reopen rate, customer effort, exception count, or review quality beside it. State what movement might mean and what investigation is required before action. Correlation and trend do not by themselves identify cause.

Version the definition. Record the effective date, change reason, approver, expected break in comparability, backfill decision, and dashboards or decisions affected. A renamed metric is not necessarily comparable if its population or formula changed. Reopen approval after changes to the workflow, source schema, event instrumentation, AI contribution, reporting purpose, or consequence.

Sources: [1] [2] [3]

Metric definition record

| Field | Required content | | --- | --- | | decision | user, action, cadence, material error | | construct | input, activity, output, quality, outcome, risk, or balance | | population | entity, event, period, inclusion, exclusion | | numerator | exact qualifying event and grain | | denominator | exact eligible population and grain | | calculation | formula, units, filters, joins, rounding, nulls | | lineage | sources, fields, transformations, extraction time | | quality | reconciliation, completeness, validity, latency, restatement | | interpretation | direction, threshold, context, limitations, prohibited uses | | governance | owner, reviewer, version, effective date, change trigger |

The record is complete only when an implementer can reproduce the metric and a reader can explain when not to use it.

Worked example: synthetic first-pass resolution rate

A fictional service team asks an AI assistant to define “first-contact resolution.” The first suggestion divides cases closed on the opening day by all created tickets. It sounds plausible but mixes speed with resolution, includes spam, treats automated acknowledgements as contact, and has no rule for reopened cases.

Using SPEC, the team changes the decision to a monthly investigation trigger for one support workflow. The unit is a support case, not a message. Eligible cases are human-routed cases closed in the month, excluding spam, test records, merged duplicates, and cases awaiting customer information. A case counts as first-pass resolved when it receives one substantive human response, closes without transfer, and has no reopen event during the next seven complete days.

The February source contains 1,260 created tickets. After exclusions and the seven-day maturity rule, 1,080 cases are eligible. Of those, 756 meet the first-pass rule, producing 70.0%. A naïve same-day calculation would have reported 78.4%. Review finds that 42 cases were reopened after an initial closure and that 68 end-of-month cases were not yet mature. The definition prevents both from being hidden.

The dashboard shows the rate beside reopen rate and median customer wait. The record states that the measure supports workflow investigation, not individual performance evaluation. A channel migration changes the case identifier in April, so the owner records a new version and runs a parallel reconciliation before comparing periods. This synthetic example demonstrates a reproducible definition; it does not establish that the metric improves service outcomes.

Two Prune boundary

Two Prune can assess whether a participant defines a decision-linked metric, resolves population and formula ambiguity, tests data quality, and communicates interpretation limits in a realistic simulation. It reports observed assessment evidence. It does not operate a customer's analytics system, certify metric accuracy, predict workplace outcomes, or make employment decisions.

Sources

Government Digital Service: Using performance data to improve your service

  • Department for Business, Innovation, Science and Trade: Digital-service indicators methodology
  • Department for Energy Security and Net Zero: Monitoring and evaluation framework

Sources: [1] [2] [3]

Explore this topic