AI Measurement Templates for Evidence-Led Programmes

Template 1: metric contract

Record business question, construct, numerator, denominator, population, window, source fields, refresh schedule, owner, exclusions, missingness, privacy rule, and decision. Add a worked calculation from a sample record. Version the definition and mark affected periods when instrumentation changes.

The U.S. GAO data-reliability guide focuses on accuracy, completeness, and applicability for the intended purpose (GAO data reliability). NIST’s AI RMF Core supports context-specific metrics, risk measurement, human oversight, and documented limitations (AI RMF Core). These sources support the template structure, not any particular result.

Sources: [1] [2]

Template 2: construct-to-evidence map

Use rows for access, activity, workflow integration, applied capability, artifact quality, risk, and outcome. For each, list the evidence source, what it can show, what it cannot show, opportunity requirement, reviewer, and missingness rule. This prevents prompt counts from becoming a capability measure.

Template 3: experiment card

Record hypothesis, unit, eligibility, assignment or comparison method, intervention, outcome, observation window, sample assumptions, contamination risks, analysis plan, stop rule, owner, and decision. Predefine how exclusions and missing data will be handled. Label synthetic or exploratory designs honestly.

Worked measurement pack

Consider a synthetic post-training pilot with 30 volunteers. The question is whether the exercise reveals verification behaviors worth addressing in curriculum. The metric contract defines completion only within the task. The evidence map distinguishes source-opening events, final artifact corrections, and reviewer judgment. The experiment card records voluntary selection and therefore limits cohort inference.

The qualitative record links interview findings to exact excerpt IDs and keeps frequency within the reviewed set. The interpretation note says the evidence can inform scenario and curriculum changes but does not establish workplace transfer or business impact.

A failed pack reports “80% AI readiness” by combining task completion and confidence. The repaired pack keeps constructs separate.

Template 4: qualitative finding record

Record question, participant context, excerpt IDs, descriptive code, analytical theme, contrary evidence, reviewer, affected decision, and next inquiry. Do not convert a small purposive interview set into a population percentage.

Template 5: interpretation note

State finding, evidence, confidence rationale, missingness, alternative explanations, boundary, and next decision. Separate observed result from inference and hypothesis. Record what evidence would change the interpretation.

Template 6: measurement change log

Record the date, metric version, changed field, reason, affected periods, backfill decision, approver, and comparison warning. Link the old and new definitions. If an instrumentation change alters coverage, do not silently splice the series. Show a break, reprocess earlier records when defensible, or limit the comparison window.

For a synthetic example, imagine that a workflow tool begins recording source openings only after a desktop update. The change log marks earlier source-use rates as unavailable, not zero. The interpretation note limits any trend statement to the post-update period, while the evidence map retains reviewer observations as a separate signal. This small artifact prevents a technical telemetry change from being reported as sudden behavior change.

Schedule a periodic owner review for definitions and access. Retire unused metrics rather than preserving a dashboard solely for continuity. Archive the rationale so future analysts can reproduce why a measure changed.

Verification checklist:

The pack supports disciplined measurement. It does not create validated benchmarks, causal evidence, or general capability scores by itself.

• Every measure answers a named question.
• Definitions and denominators are reproducible.
• Evidence sources match the construct.
• Missingness and instrumentation changes are visible.
• Quantitative and qualitative evidence retain their limits.
• Privacy and access are reviewed.
• Interpretation does not exceed the design.