Measuring role-specific AI capability in knowledge work

Start from work, not a job title

Collect recurring tasks from current role documentation, work samples, managers, and practitioners. Describe the action, inputs, audience, decision, output, tools, time, and consequence. Titles vary across organizations, and two people with the same title may perform very different work. Group tasks only when they preserve the same decision and evidence demands.

For each task, identify where AI can assist and where human ownership remains. Map opportunities to frame the assignment, find evidence, orchestrate AI, verify output, synthesize tradeoffs, and communicate the result. Exclude tasks that cannot be simulated without sensitive data or that would create an inappropriate consequence.

The US Department of Labor-sponsored ONET Content Model distinguishes occupational tasks, detailed work activities, work context, skills, knowledge, and software use. It can provide a structured starting vocabulary, but local work analysis is still required because ONET does not define a company's current AI-enabled workflow.

Sources: [1]

Construct the role-task evidence map

Each map entry should record the task, trigger, inputs, source authority, constraints, intended output, audience, AI permission, principal failure, observable behavior, evidence artifact, reviewer rule, missingness condition, and change trigger. Include the functional owner who can confirm realism and the reviewer who can explain the evidence.

Avoid one generic scoring surface pasted across roles. Verification in a market brief may involve population and date checks. Verification in an operations plan may involve permissions, dependencies, and handoffs. Shared capability labels can support comparison only when the task-specific behaviors and opportunities remain visible.

The NIST AI RMF Core includes outcomes for application scope, operator and practitioner proficiency, human oversight, context mapping, and documentation. It supports context-specific measurement without prescribing a universal role score.

Sources: [2]

Synthetic map: customer implementation work

A synthetic implementation consultant must turn an inbox escalation, configuration extract, support policy, and rollout spreadsheet into a client recovery plan. The map identifies task framing, evidence conflict resolution, AI-assisted drafting, permission verification, judgment about sequence, and communication of owners and dates.

The scenario includes a customer request that conflicts with the support policy and a spreadsheet row owned by the wrong team. A participant who produces a fluent plan without resolving either conflict shows different evidence from a participant who cites both, proposes a permitted fallback, and assigns an owner for clarification. The map retains those distinctions instead of treating both plans as equally complete.

A sales-operations analyst may also create a recovery plan, but the source systems, authority, audience, and principal failures differ. The organization creates a separate map only when those differences change the task decision and evidence required. This example is synthetic and does not validate a role benchmark.

Verify role relevance and interpretation

Ask current practitioners and a functional owner to complete or review the task using the intended tools and time. Check that source materials are realistic, expected behaviors have fair opportunities, and the rubric does not reward irrelevant polish. Test accessibility and technical failure routes.

Have independent reviewers explain their dispositions from the artifact and event evidence. Report missing opportunities, reviewer disagreement, scenario constraints, and confidence limitations. Reopen the map when the role, workflow, tools, source systems, policy, authority, or quality standard changes.

Two Prune can create controlled opportunities to observe selected role-relevant AI-work behaviors and present task-bounded evidence for human interpretation. It does not establish universal competence, mature occupational norms, future workplace performance, or employment decisions.