Practical guide
How to create an AI learning needs analysis
An AI learning needs analysis should identify the work people must perform, the evidence of current capability, the risks that require control, and the smallest learning intervention worth testing. It should not begin with a catalogue of generic AI courses or a survey asking employees what topics interest them. Use TRACE: Tasks, Required judgment, Actual evidence, Constraints, and Experiment..
An AI learning needs analysis should identify the work people must perform, the evidence of current capability, the risks that require control, and the smallest learning intervention worth testing. It should not begin with a catalogue of generic AI courses or a survey asking employees what topics interest them. Use TRACE: Tasks, Required judgment, Actual evidence, Constraints, and Experiment. The output is a decision record that connects a real work requirement to an observable capability gap and a proportionate response.
1. Define the decision before collecting data
Write the decision in one sentence: “Decide whether the customer-operations team needs practice in verifying AI-assisted policy answers before the new knowledge tool is introduced.” Name the owner, affected group, work context, evidence cutoff, constraints, and next review date.
Do not use “improve AI skills” as the decision. AI-related work may require digital fluency, data interpretation, domain judgment, source checking, communication, or technical expertise in different proportions. The OECD's 2026 policy brief on AI and skills notes that advanced AI skills concern a small share of workers while broader digital, data, managerial, and human skills remain important. Treat that as a population-level observation, not proof of any local gap. OECD: AI and skills
Sources: [1]
2. T: Map tasks, not role labels
Select three to seven recurring or consequential tasks. Decompose each into trigger, inputs, actions, decisions, output, recipient, tools, and failure consequences.
| Task | Trigger and inputs | Critical decision | Evidence of a good output | Material failure | | --- | --- | --- | --- | --- | | Answer a policy exception query | customer question, policy library, case facts | which rule and exception control | source anchors, explicit uncertainty, correct escalation | confident answer omits the exception | | Summarize a weekly service pattern | case notes, tags, volume data | which pattern is decision-relevant | reproducible calculation and representative examples | aggregate hides a small high-risk segment | | Draft a customer update | approved facts, audience, channel | what can be stated and what remains unresolved | traceable claims, clear next action | invented cause or commitment |
Observe the task in context when possible. Interviewing only a manager can miss workarounds, tacit checks, interruptions, and tool limitations. Record whether evidence came from observation, work samples, interviews, system data, policy, or assumption.
3. R: Specify required judgment and controls
For each task, describe the capability as observable behavior. Replace “good prompting” with “supplies the relevant context, tests the result against source material, and revises or rejects unsupported output.” Replace “responsible AI” with the exact boundary, data rule, verification step, or escalation action required in that workflow.
NIST's AI Risk Management Framework separates mapping context, measuring risk, managing responses, and governing responsibility. It also calls for defining and assessing operator proficiency and human oversight in context. Use those outcomes to prompt questions about the work, not as a universal curriculum. NIST AI RMF 1.0
Create one row per requirement:
| Requirement | Observable behavior | Conditions | Minimum evidence | Who judges it | | --- | --- | --- | --- | --- | | source reconciliation | finds conflicting clauses and explains which controls | timed case with two versions | final answer plus source trail | trained domain reviewer | | AI delegation | gives the tool bounded work and retains the decision | approved assistant available | interaction record and deliverable | assessment reviewer | | uncertainty handling | distinguishes missing evidence from a negative conclusion | incomplete case | explicit limitation and next check | domain reviewer |
Sources: [2]
4. A: Gather actual capability evidence
Use at least two evidence types and label their limits.
Freeze the sample, task version, scoring anchors, missingness rules, and evidence period before drawing conclusions. Separate “not observed” from “performed poorly.” If a required signal was not captured, record insufficient evidence.
- Work samples show outputs but may not reveal the process or whether assistance was used.
- Structured observation shows behavior in context but can change behavior and is hard to compare.
- A realistic work simulation can hold task conditions constant and capture process evidence, but transfer to everyday work is not automatic.
- System activity can show use patterns but not capability or judgment by itself.
- Self-report can reveal confidence, access, and perceived barriers but should not be treated as demonstrated performance.
- Manager judgment can add context but needs defined anchors and calibration.
5. C: Diagnose constraints before prescribing learning
A performance gap is not always a learning gap. Test five alternatives:
Record evidence for and against each explanation. If source documents are outdated, training people to search them faster will not solve the problem. If the escalation route has a three-day delay, a learner may knowingly accept risk because the operating design leaves no practical alternative.
- Access: people lack the approved tool, source, permission, or time.
- Clarity: the expected method, decision right, or quality rule is unclear.
- Workflow: the interface or process makes the correct action difficult.
- Incentive: speed, volume, or local norms reward skipping verification.
- Capability: people cannot yet perform the required behavior under realistic conditions.
6. E: Choose the smallest useful experiment
Match the response to the diagnosed cause. Options include a worked example, practice case, manager calibration, job aid, workflow change, source repair, access change, coached repetition, or a longer programme. State what would make the intervention continue, change, or stop.
| Field | Decision record | | --- | --- | | target behavior | reconcile conflicting source rules before answering | | intervention | 35-minute practice case plus source-conflict checklist | | comparison | alternate but equivalent case before and after practice | | primary evidence | source trail, final rationale, reviewer anchors | | confounds | prior familiarity, task-form difference, missing interaction data | | continue rule | reviewers can apply anchors consistently and the task reveals distinct approaches | | repair rule | critical evidence missing in more than 10% of sessions | | stop rule | task teaches a trick rather than the intended behavior |
The analysis recommends a test, not a promised outcome. A result from one team, task, and period should not be generalized without further evidence.
Worked example: customer-operations verification
A customer-operations director requests “AI training” after two incorrect policy answers reach customers. TRACE begins with 42 recent cases, interviews with six agents and two reviewers, source-version logs, and a short synthetic case.
Task mapping shows that agents can draft clear responses. The failure appears when a general rule conflicts with a later exception. In the sample, 11 cases include a conflict; four answers omit the exception. Two of those four were drafted without AI and two with AI. The evidence therefore does not support an AI-use-only explanation.
The requirement is narrowed to source reconciliation under time pressure. Observation finds three constraints: search ranks the older general policy first, the exception lacks a cross-link, and the escalation route is unclear. The synthetic case also shows mixed capability: five of twelve participants identify the conflict, four cite only the general rule, and three have insufficient interaction evidence because the capture tool fails.
The owner approves three actions: repair the source link, publish an escalation rule, and test a practice case with frozen anchors. The learning team does not launch a broad prompt-writing course. After the source repair, a new task form will test whether participants locate, compare, and explain the controlling exception. Results will be reported as evidence for that task and sample, with missing sessions visible.
Acceptance checklist
Two Prune can provide a realistic assessment simulation and an evidence packet showing how a participant frames a task, navigates sources, uses AI, verifies claims, applies judgment, and communicates a deliverable. It does not diagnose a workforce by itself, establish mature norms, promise training outcomes, or make employment decisions. Programme owners remain responsible for the needs analysis, interpretation, and intervention decision.
- [ ] The analysis names a real decision and owner.
- [ ] Tasks are described through inputs, decisions, outputs, and failures.
- [ ] Requirements are observable and tied to the work context.
- [ ] At least two evidence types are used with limitations recorded.
- [ ] Missing evidence is not scored as poor performance.
- [ ] Access, clarity, workflow, and incentive constraints are tested.
- [ ] The response is proportionate to the diagnosed cause.
- [ ] The experiment has frozen measures, confounds, and decision rules.
- [ ] Claims are limited to the sample, task, and evidence period.
- [ ] A review date and accountable next action are recorded.
Sources
OECD: AI and skills
- OECD: Artificial intelligence and the changing demand for skills in the labour market
- NIST: Artificial Intelligence Risk Management Framework 1.0
Sources: [1] [3] [2]
Sources
- 1.OECD: AI and skills · www.oecd.org
- 2.NIST AI RMF 1.0 · National Institute of Standards and Technology
- 3.OECD: Artificial intelligence and the changing demand for skills in the labour market · www.oecd.org