How to measure workflow evidence in AI-enabled work

Draw an evidence coverage graph

Start with the task standard, not the event stream. For each consequential decision, connect four nodes: opportunity, observable action, retained evidence, and permissible interpretation. The NIST AI RMF Measure function emphasizes documenting metrics, monitoring, and the limits of measurement. The NIST AI RMF Core likewise treats measurement as one part of a broader governance process.

Suppose a procurement analyst must recommend one of three vendors. A coverage graph could include:

The same trace does not establish a stable trait, explain why an action was omitted, or show performance in every procurement context.

• Opportunity: notice that implementation dates conflict across a proposal and its appendix.
• Observable action: open both sources, annotate the conflict, and ask AI to separate confirmed from unconfirmed dates.
• Retained evidence: source-open events, saved note, prompt and response, and revised comparison table.
• Permissible interpretation: the analyst detected and handled this contradiction in this simulation.

Sources: [1] [2]

Measure decisions, artifacts, and transitions

Use three evidence layers. Decision evidence captures the choice made and the basis stated for it. Artifact evidence captures the table, memo, calculation, or other deliverable. Transition evidence captures meaningful changes between states, such as replacing an unsupported claim after reviewing a source.

Avoid scoring every click. Opening a document may indicate access, curiosity, confusion, or an accidental action. A stronger indicator combines an opportunity with a relevant action and a visible consequence in the artifact. For example, a revised total tied to an identified unit mismatch carries more interpretive weight than a spreadsheet-open event alone.

In the synthetic vendor task, the evidence bundle might contain a source matrix, two AI exchanges, a calculation revision, a note on missing security documentation, and the final recommendation. Each item should have a timestamp and stable identifier so reviewers can reconstruct the sequence without relying on memory.

Audit missingness before scoring

Create a missingness register with four causes: the task never offered the opportunity, the participant did not act, the product failed to capture the action, or the evidence cannot be interpreted confidently. These conditions must not be collapsed into the same zero.

Run a replay test before using a metric. Give a reviewer the task inputs, retained trace, and final artifact. Ask whether the reviewer can locate the claimed behavior and whether another plausible explanation remains. Then compare the trace with the original system record to identify dropped or duplicated events.

The NIST Information Quality Standards provide a practical reminder that information quality includes utility, objectivity, and integrity. Applied here, a technically valid log may still be unhelpful if it cannot answer the decision for which it is being collected.

Sources: [3]

Use an evidence coverage table

A small table keeps the measurement claim inspectable:

Reviewers should record disagreements and the evidence that resolved them. If disagreement persists, narrow the claim or mark the signal as uncertain rather than creating false precision.

• Source reconciliation: two opportunities, one handled, one not observed; evidence is a note plus table revision; interpretation confidence is moderate.
• AI critique: one opportunity, critique requested, recommendation changed; evidence is the exchange and version diff; confidence is high for this instance.
• Risk communication: one opportunity, unresolved security evidence disclosed in the memo; confidence is high.
• Calculation verification: no meaningful opportunity because the supplied totals were already reconciled; do not score absence as failure.

Keep measurement tied to its decision

Before release, write the consumer, decision, time horizon, and consequence of error for every metric. Retain only the minimum trace needed for that purpose, with access and retention controls appropriate to the data. Revisit the coverage graph when the task, tool, or capture system changes.

Two Prune can collect workspace activity and submitted artifacts in a realistic assessment, then present evidence supporting observed performance. It does not turn event volume into a universal capability score, infer motives from clicks, or make employment decisions. Workflow evidence is strongest when the task created a fair opportunity, the trace is intact, and the interpretation stays within the observed scenario.