Measuring cohort capability in AI-enabled knowledge work

Define the cohort and the decision

State who is included, why they belong together, the observation period, the task family, and the decision the analysis will inform. A learning team deciding which practice exercise to offer next needs different evidence from a programme owner checking whether a common foundation task is understood.

Define observable capabilities before inspecting results. Possible dimensions include framing, source use, AI instruction, iteration, checking, judgment, escalation, and delivery. Preserve task difficulty, time limit, tool access, source environment, and language because these conditions affect comparability.

Skills England's 2026 AI foundation skills for work benchmark groups basic workplace AI skills across technical, non-technical, and responsible or ethical domains. It applies in England and concerns foundation skills; it can inform a construct map but does not validate a specific assessment or prove workplace transfer.

Sources: [1]

Construct the opportunity-adjusted matrix

Create one row per participant and columns for cohort membership, task version, capability opportunity, evidence captured, observable anchor, result, missingness reason, reviewer confidence, accommodation or condition, and follow-up need. Aggregate only cells with sufficiently comparable opportunity and evidence.

Report denominators beside every result. “Sixty percent verified outputs” is incomplete without the number observed, the verification opportunity, and what counted as evidence. Separate not demonstrated, not observed, not elicited, and not scorable. Do not convert missing opportunity into a zero.

Stratify by task version or relevant operating condition before interpreting differences. Small subgroups may be described without publishing unstable percentages. Where reviewers score samples, examine agreement and ambiguous anchors before treating a group difference as participant capability.

Add an evidence-coverage view beside the capability view. For every capability, show how many people received a comparable opportunity, how many produced usable evidence, and how many records were limited by capture or scoring problems. This prevents a clean-looking group average from masking a task-design failure. Keep accommodations and contextual conditions visible without treating them as automatic explanations for performance.

The 2026 UK employer guide to AI upskilling emphasizes contextualized, inclusive, and sustained workforce capability building. Its research can support task-relevant learning design, but local cohort outcomes and transfer still require direct evidence.

Sources: [2]

Worked artifact: a synthetic enablement cohort

Imagine 18 operations staff complete one of two synthetic service-analysis tasks. Version A contains conflicting manager notes; version B does not. Twelve participants receive a genuine opportunity to reconcile contradictory evidence, while six cannot demonstrate that behavior because their task lacks the conflict.

The matrix shows that eight of the twelve identify and resolve the conflict with source evidence. The other six are marked “not elicited,” not unsuccessful. A separate calculation-checking opportunity exists in both versions, but two event logs are incomplete, so the report gives the observed denominator and flags capture missingness.

The learning team selects a follow-up exercise on conflict resolution and repairs logging before the next cohort. It does not rank individuals or infer broad workforce readiness. The example is synthetic and supplies no measured result for a real programme.

Verify the cohort inference

Recalculate aggregates from row-level evidence and check inclusion rules, denominators, task versions, missingness, and rounding. Have a second reviewer rescore risk-selected boundary cases. Test whether an apparent subgroup difference disappears after controlling for task opportunity or evidence capture.

Compare over time only when the construct, task conditions, anchors, and capture method remain sufficiently stable. If they changed, present separate slices and explain the break. A larger sample does not repair a mismatched task or invalid inference.

Two Prune can provide task-bounded assessment evidence for human interpretation. Cohort reporting should state evidence strength, confidence limitations, and follow-up questions. It does not establish universal capability norms, predict individual outcomes, or make employment decisions.