How to develop and assess success criteria in AI-enabled work

Define success before selecting a metric

GOV.UK service guidance recommends setting performance measures from the start, linking them to service objectives, and gathering the data needed to understand performance. NIST's AI Risk Management Framework says mission and relevant goals should be understood and documented, measurement approaches should connect to the deployment context, and limits or risks that cannot be measured should also be recorded.

These principles matter for everyday knowledge work. A request such as “improve the briefing” is not yet a success definition. Neither is “use AI to save time.” A useful criterion connects a desired outcome to observable evidence and a decision. It also protects against proxy success, where a visible number improves while the work becomes less useful, less accurate, or harder to review.

Metrics are inputs to judgment. They do not replace the need to state the audience, conditions, trade-offs, and uncertainty around the work.

Build a SIGNAL criteria card

Use a SIGNAL card before producing the artifact:

| Field | Criteria question | Evidence artifact | | --- | --- | --- | | Stakeholder | Who experiences the outcome or makes the decision? | Named audience and decision owner | | Intended outcome | What useful change should the work enable? | Outcome statement, not an activity count | | Gauge | Which observable indicator would show progress or completion? | Measure, evidence source, and collection method | | Necessary guardrails | What must not be sacrificed to improve the gauge? | Accuracy, risk, accessibility, cost, or policy limits | | Acceptance range | What result is sufficient, insufficient, or ambiguous? | Threshold, rubric, or decision rule with rationale | | Learning plan | When will the criterion be reviewed, and what would cause revision? | Review date, owner, and trigger |

Use more than one indicator only when each answers a different question. Label leading signals, such as completion of a review step, separately from outcome signals, such as whether a manager can correctly apply a policy. Record missing data and conditions where the measure would be misleading.

AI can suggest indicators and edge cases, but the worker should test whether the proposed measure is observable, attributable enough for the decision, and resistant to obvious gaming. A metric that is easy to count is not automatically relevant.

Worked example: a synthetic incident-briefing workflow

Consider a synthetic support team redesigning a daily incident briefing. The request says the briefing should be “faster and more useful,” and the team plans to use AI to summarize tickets. This example is original and does not describe a Two Prune customer.

The first draft criterion is “generate the briefing in under five minutes.” That measures production speed, but it does not show whether the briefing supports the duty manager's decision. The team creates a SIGNAL card:

Generation time remains a secondary operational measure. It matters only after the decision artifact passes the critical coverage, traceability, and boundary checks. A five-minute briefing that hides an unresolved conflict fails. A seven-minute briefing may reveal a workflow bottleneck without making the artifact itself unsuccessful.

The team builds a small evidence table linking each criterion to a test, result, reviewer, and open limitation. That table is more useful than a single “quality score” because it shows which decision conditions passed and which remain uncertain.

• Stakeholder: the duty manager deciding which incidents require escalation before the morning handover;
• Intended outcome: identify priority incidents, unresolved evidence, and accountable next actions in time for the handover;
• Gauge: in a fixed test set, the manager can recover each priority incident, its current evidence status, owner, and next decision from the briefing;
• Necessary guardrails: no invented status, every material statement traces to a ticket or approved update, restricted details stay out of the general summary, and unresolved conflicts remain visible;
• Acceptance range: all critical incidents are present and traceable, with no unsupported status statements; lower-priority omissions are reviewed separately rather than averaged into the critical check;
• Learning plan: review after two synthetic test rounds and revise the format if managers repeatedly misread ownership or uncertainty.

Practice with proxy swaps

Give a learner a vague objective and three tempting proxies. For a training debrief, the proxies might be attendance, number of AI prompts, and participant satisfaction. Ask the learner to build a SIGNAL card and explain what each proxy can and cannot establish.

Then change one condition. For example, the audience shifts from course facilitators to functional managers, or a policy requires evidence of source verification. The learner should revise the intended outcome, guardrails, acceptance rule, or collection method rather than merely adding another metric.

Useful practice includes:

• rewriting an activity target as an outcome for a named stakeholder;
• selecting the smallest set of indicators that supports the decision;
• adding a guardrail that could reveal harmful proxy optimization;
• defining ambiguous and missing-data states;
• testing the criterion on a strong, weak, and deceptive artifact;
• recording when the criterion should be revised.

Assess observable criteria-setting behavior

In an assessment, provide a brief with an aspirational objective, incomplete measures, and pressure to report a simple result. Observe whether the participant:

Do not award credit merely for using a familiar acronym or numeric target. A precise percentage can be arbitrary, and a qualitative rubric can be strong when it defines observable differences and review evidence. Evaluate the reasoning and traceability behind the criterion.

• identifies the stakeholder and decision that the criteria must support;
• distinguishes activities, outputs, outcomes, and guardrails;
• chooses evidence that can actually be collected in the task;
• explains why each indicator is relevant and where it is weak;
• defines sufficient, insufficient, ambiguous, and missing-evidence states;
• tests for proxy gaming or trade-offs hidden by an aggregate score;
• checks AI-suggested metrics against the task and evidence environment;
• updates the criteria when the decision context changes.

Verify with a three-case criterion test

Apply each criterion to three deliberately designed cases: one that should pass, one that should fail, and one that should remain uncertain because evidence is missing or conflicting. A useful criterion separates these cases for the stated reason. If all three receive the same result, the gauge, guardrail, or acceptance rule needs revision.

Next, ask an independent reviewer to reproduce the decision using the SIGNAL card and retained evidence. Compare disagreements and identify whether they arise from unclear criteria, insufficient evidence, or legitimate judgment. Preserve that limitation instead of hiding it in an average.

Success criteria are task-bounded. Meeting them does not prove broad capability or future performance, and the criteria themselves may need revision as evidence changes. In a Two Prune assessment, the criteria card, source use, AI interactions, verification, revisions, and final artifact can contribute bounded evidence about Context Framing, Verification & Risk Control, Judgment & Synthesis, and Communication & Delivery. They do not make an employment decision.

Sources

GOV.UK guidance on using performance data to improve a service

• NIST AI Risk Management Framework Core

Sources: [1] [2]