Questions to Ask Before Measuring AI-Enabled Work

Ask about the decision and user

Who will use the finding? What action is available to them? Is the decision about a tool rollout, a learning programme, a workflow, or an individual assessment? What is the cost of a false positive or false negative? A metric without a named decision often becomes a dashboard decoration or is reused for a stronger claim than it can support.

The CDC’s 2024 evaluation framework begins with assessing context, describing the programme, and focusing evaluation questions before gathering evidence (CDC approach to programme evaluation). That sequence supports decision-led measurement. It does not imply that public-health evaluation rules automatically govern an enterprise AI programme.

Sources: [1]

Ask what the measure actually observes

Does the data record access, frequency, a workflow action, a final artifact, a verification behavior, or an outcome? What relevant activity happens outside the measured system? Are tasks comparable in difficulty and assistance? Could a lower count reflect efficiency rather than disengagement? What denominator, time window, exclusion rule, and data-quality limitation must travel with the result?

GAO’s AI accountability framework separates governance, data, performance, and monitoring and provides questions for managers and assessors (GAO AI accountability framework). That separation is useful because a clean usage table does not establish work quality, and a strong controlled task does not establish enterprise impact.

Sources: [2]

Build a question-to-decision map

Use six columns: decision, primary question, construct, observable evidence, rival explanation, and allowed interpretation. Add a seventh column for the next evidence action. The map should contain at least one disconfirming question for every headline metric. For example, “Did adoption increase?” pairs with “Did the licensed population or logging configuration change?”

Worked artifact: post-training review

Imagine a synthetic AI learning programme. Attendance is 88 percent, weekly tool use increased, and participants report greater confidence. The accountable owner wants to know whether to continue the course. The map records that attendance answers participation, telemetry answers observed use in one system, and confidence answers self-perception. None directly answers whether people verify evidence in realistic work.

The team adds a small controlled simulation with a contradictory source and a calculation trap. The evidence shows which checks occurred and what was delivered, with missingness labels. The continuation decision combines participation, observed behavior, manager context, cost, and limitations. It does not turn the simulation into proof of business impact.

Review the inference chain

Before reporting, read each conclusion backward: decision, interpretation, measure, raw observation, collection process. Mark every step that depends on an assumption. Test whether another plausible explanation fits the same pattern. If evidence is insufficient, report the limitation and the smallest useful next check rather than manufacturing certainty.

Set a retirement rule for every measure

A measure should have an owner, review date, and condition that makes it unsuitable. Logging changes, population shifts, new workflow stages, and revised definitions can break a time series without producing an obvious error. Preserve the old definition and explain any bridge rather than presenting unlike periods as continuous.

Also ask who is missing from the data. People may use other tools, work offline, or avoid a system because the task is unsuitable. Missingness is a property of the evidence collection process, not automatically a sign of low adoption or low capability. Record what cannot be seen before acting on what can.

These questions are educational prompts. They do not validate a metric, establish causation, or create mature benchmarks. Individual assessment evidence should remain task-bounded, confidence-aware, and interpreted by people rather than used for automated employment decisions.