How to develop and assess decision criteria in AI-enabled work

Start with the decision, not the scoring scale

A criteria set is useful only when it belongs to a named decision. Write one sentence that identifies the choice, the decision owner, the options that are genuinely available, and the time horizon. “Choose a document-review approach for the next six-month pilot” is testable. “Find the best AI tool” is not, because neither best nor the operating context is defined.

The UK government's guidance on multi-criteria decision analysis describes MCDA as a way to appraise options against multiple criteria and warns that its value depends on sound problem structuring, transparent judgments, and appropriate handling of uncertainty. The World Health Organization's Evidence-to-Decision framework likewise uses explicit criteria so that important factors are considered and disagreements can be made visible. Neither framework turns judgment into arithmetic. They make the judgment legible.

Before listing preferences, identify gateway conditions. A gateway is a pass-or-stop requirement, such as compatibility with an approved data environment or the ability to provide an auditable export. An option that fails a real gateway should not receive compensating points for attractive but optional features. Calling every preference a gateway, however, eliminates comparison and can conceal who imposed the constraint.

Build a CLEAR criteria card

Use a CLEAR card to define each criterion before any option is scored:

| Field | Question | What to record | | --- | --- | --- | | Consequence | What decision outcome would this criterion protect or improve? | A concrete consequence, not a feature label | | Level | Is it a gateway, a weighted comparison, or a tie-breaker? | One role in the decision | | Evidence | What observable record would show how an option performs? | Test result, policy record, cost model, or user trial | | Anchor | What do low, acceptable, and strong performance mean? | Behavioural or numerical descriptions | | Responsibility | Who owns the judgment and who may challenge it? | Named decision role and review point |

This structure catches several common defects. “Ease of use” is too vague until the team specifies a consequential task, an evidence source, and anchors. “Security” may improperly combine a gateway compliance check, a weighted operational-risk judgment, and a tie-breaker about administrative burden. Split a criterion when its components require different evidence or decision rules. Do not split it merely to make the scorecard look detailed.

Criteria should also be reasonably independent. If “setup effort,” “implementation time,” and “speed to launch” all use the same onboarding test, counting all three may triple-weight one concern. Map every criterion to its evidence before assigning weights. Duplicate evidence is a prompt to combine criteria or explain why separate consequences justify separate judgments.

Weights express priorities for this decision, not universal truths. Ask the decision owner to allocate importance before seeing option scores where practical, record the rationale, and run a sensitivity check. If a small plausible change in weights reverses the result, the correct output is a fragile preference that needs further evidence or an explicit judgment, not a confident winner.

Worked artifact: selecting a review workflow

Consider a synthetic operations team choosing between three approaches for reviewing AI-assisted client briefs during a six-month internal pilot:

This example is original and synthetic. It does not describe a Two Prune customer, product result, or benchmark.

The team first writes its decision sentence: “The operations lead will choose one review workflow for the six-month pilot, using the existing approved document environment and a maximum of two hours of reviewer training.” The approved environment and training limit become gateways. Option C cannot be evaluated until its environment is approved, so it is marked ineligible rather than given a low score that other benefits could offset.

For A and B, the team creates four CLEAR cards:

The team does not ask an AI assistant to award the final score from descriptions. It uses AI to challenge overlap between criteria, draft counterexamples for the anchors, and summarize test notes. Humans inspect the source records and assign scores with written rationales.

Option B leads under the initial weights because its required fields improve traceability and exception recovery, while Option A requires less routine effort. A sensitivity test increases the effort weight from 20 to 35 and proportionally reduces the first two weights. B still leads, but by a narrow margin. The final decision record therefore says B is preferred for the pilot, names the effort risk, and requires an effort check after the first two weeks. That is more useful than presenting a decimal total as objective proof.

• Option A: a manual checklist in the current document system;
• Option B: a structured review form with required evidence fields;
• Option C: a new external workflow service.
• Review completeness. Consequence: critical claims are less likely to leave review without an evidence note. Evidence: ten seeded briefs containing known omissions. Anchors: low means fewer than six omissions are identified; acceptable means six to eight; strong means nine or ten. Weight: 35.
• Traceability. Consequence: another reviewer can locate the reason for approval or return. Evidence: audit of the completed record. Anchors describe whether claim, source, reviewer action, and date are recoverable. Weight: 30.
• Routine effort. Consequence: the workflow fits the review capacity. Evidence: timed use across the same ten briefs, with interruptions noted. Anchors use ranges agreed before testing. Weight: 20.
• Exception recovery. Consequence: ambiguous or unsupported claims reach the right owner. Evidence: three seeded exceptions and the resulting handoff record. Anchors describe whether ownership and next action are explicit. Weight: 15.

Practice the skill through criteria debugging

Give a learner a short decision brief, an evidence pack, and a flawed scorecard. Seed defects that require judgment rather than proofreading: a hidden gateway, two duplicate criteria, one criterion with no observable evidence, anchors that mix speed and quality, and weights that were chosen after the preferred option was known.

Ask the learner to produce a decision sentence, repaired CLEAR cards, an evidence map, and one sensitivity test. AI may help generate alternative anchors or identify where phrases are ambiguous. The learner should accept, revise, or reject those suggestions and explain why. This reveals whether the person can control the comparison instead of merely formatting model output.

Useful practice varies the consequence of error. A low-stakes choice may need a few plain-language tests. A consequential decision with uncertain effects may need specialist input, multiple scenarios, and documented dissent. More rows are not automatically more rigorous.

Assess the observable work

Assess the process and artifact against evidence such as:

Retain the original and revised cards, evidence map, test records, weight rationale, sensitivity result, and decision note. A polished matrix alone is weak evidence because it can hide how the criteria were chosen and whether the scores were grounded.

• decision framing: the owner, options, scope, time horizon, and constraints are explicit;
• criterion validity: each comparison criterion connects to a relevant consequence;
• gateway discipline: non-negotiable conditions cannot be offset by unrelated benefits;
• anchor quality: a reviewer can distinguish low, acceptable, and strong performance from evidence;
• double-count control: overlapping criteria and reused evidence are identified;
• trade-off reasoning: weights and tie-breakers have recorded rationales;
• uncertainty handling: sensitivity, missing evidence, and unresolved disagreement are visible;
• revision behaviour: the participant repairs the criteria when a counterexample exposes a defect.

Verify the criteria before using the result

Run four checks. First, remove option names and ask whether the criteria still describe the desired consequences. If not, the scorecard may have been written to favour a known option. Second, trace every score to the promised evidence and anchor. Third, vary uncertain weights and borderline scores within plausible ranges. Fourth, ask what new evidence would change the decision.

Decision criteria support structured judgment. They do not prove that an option will deliver an outcome, erase value choices, or replace the accountable decision owner. In a Two Prune assessment, the exercise can provide evidence about Judgment & Synthesis, Evidence Navigation, and Verification & Risk Control within the simulation. It produces an assessment signal about observed work, not an employment decision or a prediction of future performance.

Sources

UK Government: Multi-criteria decision analysis guidance

• World Health Organization: Evidence-to-Decision tables and framework
• UK Government: Stage 5 options appraisal

Sources: [1] [2] [3]