Practical guide

AI capability measurement guide for the future of work researcher

A future of work researcher needs a research decision agenda, not a generic list of AI skills. The useful question is which bounded evidence can distinguish changing task demands, AI exposure, adoption, work redesign, and demonstrated human capability without turning one measure into a claim about all of them. The researcher may own study design, influence programme choices, review evidence, or.

By Two Prune

A future of work researcher needs a research decision agenda, not a generic list of AI skills. The useful question is which bounded evidence can distinguish changing task demands, AI exposure, adoption, work redesign, and demonstrated human capability without turning one measure into a claim about all of them. The researcher may own study design, influence programme choices, review evidence, or collaborate with workforce leaders. Do not assume they own budget, operational policy, or participant decisions. This guide uses the MAPS method: mandate, alternatives, proposition map, and study gate. Its output is a copyable decision-and-evidence agenda that keeps constructs, sources, missingness, and use boundaries visible.

M: Define the research mandate

Write the decision that the research must support. Examples include selecting measures for a longitudinal study, deciding whether a controlled work simulation adds information beyond surveys, identifying tasks for qualitative inquiry, or testing whether a training intervention changes observed work in a bounded scenario.

Classify the researcher’s role for that decision:

Name the population, task domain, time horizon, unit of analysis, and prohibited uses. “Understand AI capability” is too broad. “Test whether a 30-minute synthetic source-review task adds interpretable evidence about verification behavior after a defined training programme” is bounded enough to design.

The OECD’s 2026 AI exposure measure maps AI capabilities across cognitive, social, and physical domains to occupational requirements. It presents exposure as multidimensional and explicitly notes that actual effects depend on adoption, regulation, organizational change, and social choice. Use it as a transparent example of an exposure construct, not as a direct measure of a person’s demonstrated AI fluency. OECD: The OECD AI exposure measure

  • Owner: accountable for the study question, protocol, and analysis plan.
  • Influencer: provides evidence that another accountable leader uses.
  • Reviewer: challenges construct fit, methods, or interpretation.
  • Collaborator: contributes disciplinary, operational, participant, or governance expertise.

Sources: [1]

A: Separate current alternatives

Inventory existing evidence before proposing a new assessment. Each source answers a different question.

| Evidence source | What it may show | What it does not establish by itself | | --- | --- | --- | | occupational exposure index | potential relationship between AI capabilities and occupational requirements | actual adoption, work quality, or individual capability | | job-posting or task text | stated demand and changing language | performed work or causal change | | usage telemetry | system activity within captured tools | correct framing, verification, judgment, or off-system checks | | survey or self-assessment | attitudes, confidence, reported behavior | comparable demonstrated performance | | manager observation | contextual work examples | standardized opportunity or consistent interpretation | | final deliverable | output quality in the observed case | complete process, source use, or unseen checks | | controlled work simulation | performance in a known task and evidence environment | general workplace transfer or future performance |

Record access, coverage, selection, missingness, sensitivity, and update cadence for each source. A new measure is justified only when the decision has a material evidence gap that the measure can plausibly address.

OECD’s earlier future-of-skills work notes that direct measures rarely stand alone: expert judgment is still needed to choose relevant measures, describe their limits, and synthesize them for a broader capability. It also explains that useful evidence can change as AI capabilities develop. That is a warning against freezing a convenient benchmark and treating it as timeless. OECD: AI and the Future of Skills, Volume 2

Sources: [2]

P: Build a proposition and evidence map

For each research proposition, define the construct, opportunity, primary evidence, competing explanation, missingness rule, and interpretation boundary. Predeclare which observations would support, weaken, or leave the proposition unresolved.

| Research proposition | Construct | Opportunity required | Primary evidence | Key confound | Missingness rule | Interpretation limit | | --- | --- | --- | --- | --- | --- | --- | | participants verify AI-assisted claims against supplied sources | verification behavior in one bounded task | task contains checkable claims and accessible sources | deliverable corrections, source use, workflow evidence, reviewer trace | source ambiguity or inaccessible material | no opportunity means unavailable, not poor performance | does not prove workplace transfer |

Keep at least four distinctions explicit:

Specify evidence governance before collection. Define consent or lawful basis where applicable, access roles, retention, de-identification, participant communication, reviewer training, correction routes, and prohibited downstream uses. Individual assessment evidence should not be exposed merely because a researcher can technically retrieve it.

  • Exposure is not adoption. A task may be technically exposed while an organization has not adopted the relevant system.
  • Adoption is not capability. Use frequency does not establish whether work was framed, checked, judged, and delivered well.
  • Capability evidence is task-bounded. Performance depends on opportunity, difficulty, tools, sources, time, and support.
  • Observed association is not causation. Training participation, role selection, prior experience, accessibility, and organizational conditions may explain patterns.

S: Design the study gate

Start with the smallest study that can change the research decision. A useful pilot can compare a new controlled simulation with existing evidence for the same bounded question, while preserving uncertainty rather than seeking a flattering correlation.

The protocol should freeze:

Include a scenario-quality review before interpreting participant results. If the task fails to create a fair opportunity to demonstrate a behavior, the result is a measurement defect, not negative participant evidence. Keep output quality and workflow evidence separate enough to see where they agree or diverge.

  • construct definition and change log;
  • scenario, source packet, available tools, and time limit;
  • inclusion, exclusion, accommodation, and withdrawal rules;
  • reviewer protocol, calibration, and disagreement handling;
  • primary and secondary analyses;
  • missingness and invalid-attempt rules;
  • data access, retention, and reporting restrictions;
  • stopping, repair, and replication criteria.

Copyable decision-and-evidence agenda

| Agenda field | Required entry | | --- | --- | | decision | the research or programme choice this evidence can inform | | researcher role | owner, influencer, reviewer, or collaborator | | population and context | who, where, task family, and time horizon | | construct | observable definition and exclusions | | current alternatives | available sources and their limits | | evidence gap | what remains unknown and why it matters | | proposed measure | task, opportunity, evidence, and artifact | | alternatives and confounds | at least three plausible explanations | | missingness | unavailable, invalid, excluded, and not observed rules | | governance | access, retention, participant communication, and prohibited uses | | pilot gate | pass, repair, stop, or replicate criteria | | reporting boundary | claims permitted and claims prohibited | | owner and expiry | accountable person, review date, and change triggers |

Worked example: testing evidence after AI training

A research team is asked whether a completed AI course improved employee capability. Existing evidence includes course completion, satisfaction, self-reported confidence, and platform usage. The initial proposal is to correlate usage with a single capability score.

Mandate: the team narrows the decision to whether a controlled source-verification simulation adds useful evidence for planning a larger study. The researcher owns the protocol and analysis; a learning leader owns the programme decision; an accessibility specialist and data-governance lead are collaborators.

Alternatives: completion shows participation, confidence shows reported perception, and telemetry shows captured activity. None creates the same opportunity to inspect whether participants catch an unsupported claim in a known source environment. The simulation may add that opportunity, but it cannot establish transfer to ordinary work.

Proposition map: the primary proposition is that the simulation yields interpretable evidence about source verification for most valid attempts. The task contains two checkable claims and one source conflict. The team predeclares that missing workflow events cannot become negative evidence, accommodations are not defects, and an inaccessible source invalidates that opportunity.

Study gate: twenty volunteers complete the scenario before and after the programme, but the pilot is not powered or represented as a causal impact estimate. Two reviewers independently annotate a stratified subset. The team examines task validity, evidence completeness, reviewer disagreement, participant feedback, and whether the new evidence changes a named research decision.

The source-conflict cue proves ambiguous for six participants. Rather than label their work weak, the researchers repair the scenario and repeat a small qualification sample. The final pilot supports a larger measurement study with revised task evidence. It does not prove that the course caused workplace improvement.

Objections and boundaries

“We already have an exposure index.” Use it for the decision it was built to inform. It does not replace evidence of adoption, work design, or demonstrated performance.

“Manager observations are more realistic.” They may add context and should remain in the evidence map. They usually differ in task, opportunity, visibility, and interpretation, which limits comparability.

“A simulation is artificial.” Correct. It controls the task and evidence environment to create a consistent opportunity. Its transfer to other work remains an empirical question.

“Can we rank people?” This agenda does not authorize ranking, automated screening, or employment decisions. Early evidence may be directional, task-bounded, and unsuitable for individual comparison.

Hold the study when the construct shifts after results are visible, the task does not create the required opportunity, material missingness is hidden, access exceeds the approved purpose, reviewers cannot apply the protocol, or the requested claim exceeds the design.

Two Prune can provide structured evidence from a realistic assessment for human interpretation. It does not prove workplace performance, validated programme impact, causal change, mature benchmarks, or future job performance.

Sources

OECD: The OECD AI exposure measure

  • OECD: AI and the Future of Skills, Volume 2

Sources: [1] [2]