Measuring AI orchestration in knowledge work

Define the construct before collecting events

Start by stating what orchestration means for the task. For a research brief it may include decomposing the question, selecting sources, assigning synthesis to AI, checking citations, and resolving conflicting evidence. For a spreadsheet review it may include identifying calculation risks, asking AI to explain formulas, reperformance outside the model, and deciding which discrepancies matter.

Separate orchestration from adjacent capabilities. Finding an authoritative document is evidence navigation. Confirming a calculation is verification. Choosing among tradeoffs is judgment. Orchestration concerns how the person coordinates these activities with AI. A single action can inform several constructs, but the scoring explanation should state which evidence supports which interpretation.

The NIST AI RMF Measure guidance says measurement approaches should be selected for mapped risks and that unmeasured or unmeasurable characteristics should be documented. It also describes AI as sociotechnical, shaped by human behavior and context. This supports task-specific mixed evidence rather than treating one activity count as a universal capability score.

Sources: [1]

Build an orchestration trace

Record decision points rather than every keystroke. For each point capture the task state, available evidence, participant action, AI contribution, observed revision, verification step, and resulting change to the deliverable. Include evidence absences: a source never opened, a contradiction not tested, or a recommendation accepted without a traceable basis.

Use a small set of behavior indicators tied to the opportunity the scenario actually provided. Examples include whether the objective was reframed after new evidence, whether the participant broke a complex request into checkable parts, whether AI output was compared with the supplied record, and whether a failed approach was revised. Do not infer deliberation merely from time spent, number of prompts, or length of a transcript.

The NIST Generative AI Profile extends the voluntary AI RMF for generative AI and emphasizes context-aware risk management across design, development, use, and evaluation. It is a risk-management resource, not a validated rubric for individual performance. Its value here is in reinforcing explicit context, documented limits, testing, and monitoring.

Sources: [2]

Worked artifact: a synthetic market-entry brief

Consider a synthetic assessment in which a participant prepares a market-entry recommendation from a policy excerpt, a small dataset, two stakeholder messages, and an external source. The first orchestration-trace entry records that the participant asks AI for a recommendation before reviewing the policy. The model suggests launching in the largest market.

The participant then opens the policy, notices a distribution restriction, and changes strategy. They ask AI to compare two permitted markets, but the response cites a percentage that is not in the dataset. The participant recalculates the figures in a spreadsheet, rejects the unsupported percentage, and revises the brief to separate observed data from assumptions. A final entry shows them asking AI to critique the recommendation, checking the critique against the source pack, and retaining one valid concern.

The artifact links each material change to the evidence that triggered it. It shows productive recovery after a weak first move, rather than rewarding a polished initial prompt. The scenario and figures are synthetic and do not establish a benchmark or predict performance in another task.

Score evidence with opportunity and confidence

Define behavioral anchors before reviewing the trace. Weak evidence might show one broad request followed by unexamined acceptance. Developing evidence might show decomposition but inconsistent source grounding. Strong task-bounded evidence might show purposeful allocation, evidence-aware iteration, independent checking, and integration of corrections into the deliverable.

Adjust confidence for missing opportunity and missing telemetry. If the task contains no meaningful contradiction, it cannot strongly demonstrate challenge behavior. If activity logs are incomplete, absence of a recorded action is not proof that the participant failed to perform it elsewhere. Report the observation, alternative explanations, and the amount of evidence available.

Verify the measurement method

Have reviewers independently apply the anchors to a varied sample and compare where interpretations diverge. Trace each rating to observable events and deliverable changes. Test whether the same behavior is treated consistently across participants and whether scenario features create unequal opportunities to demonstrate it. Revisit the construct when tools, scenario design, instructions, or available telemetry change.

This approach can make orchestration evidence more interpretable. It does not establish a universally valid score, mature benchmark, or causal effect of AI use. Two Prune provides structured, task-bounded assessment evidence for human interpretation and states confidence limitations when evidence is incomplete. It does not automate employment decisions.