Evidence Questions for AI-Enabled Work Programmes

Questions about the decision

What decision will change because of this evidence? Who owns it? What is the decision date? Is the choice about curriculum, workflow design, tool access, review capacity, or a participant development conversation? What evidence would reverse the current view? If no plausible answer could change an action, the collection may be reporting rather than decision support.

GAO’s evidence-based policymaking guide organizes practice around planning for results, assessing and building evidence, using evidence, and fostering continuous learning (GAO evidence practices). That sequence is useful because it begins with intended results and use. It does not supply a universal metric for AI-enabled work.

Sources: [1]

Questions about the construct and opportunity

Are you trying to observe access, activity, workflow integration, applied capability, artifact quality, risk control, or outcome? Which of these are being combined? Does the task give each person a comparable opportunity to show the behavior? Could success arise without it? Could a capable person appear weak because an input, permission, or event was missing?

Write one observable definition and one exclusion for each construct. “Verification” might mean checking a material externally attributable claim against a suitable source before release. It excludes merely adding a citation without examining whether the source supports the claim.

Questions about source fitness

Who produced the record, for what purpose, under which definition, and during which period? Is it complete enough for the decision? Are timestamps, populations, units, and versions aligned? What transformations occurred? Can a reviewer reproduce the result? What evidence contradicts the favored explanation?

The GAO data-reliability guide frames data quality for an intended purpose through accuracy, completeness, and applicability (GAO data reliability). Use these dimensions explicitly. A complete usage export may be accurate for feature adoption while remaining inapplicable to a claim about judgment or work quality.

Sources: [2]

Worked artifact: a programme evidence map

Imagine a synthetic AI-writing programme. Leaders ask whether the course improved verification. The evidence map has one row per question. Attendance answers who joined. Tool logs answer whether a feature was used. A timed exercise creates a comparable opportunity to identify an unsupported claim. The submitted artifact shows whether the claim was corrected. Workflow records may show the source review. A manager interview describes transfer examples but is vulnerable to recall and selection effects.

The map has columns for question, construct, source, unit, window, opportunity, what it can show, what it cannot show, missingness rule, owner, and next decision. The team decides that exercise evidence can guide a curriculum revision, while workplace transfer remains an open question. It does not combine attendance and confidence into an “AI readiness” percentage.

Questions about interpretation and challenge

Which observations support the conclusion? Which oppose it? How much depends on reviewer judgment? What assumptions connect the record to the claim? Are comparisons made within similar tasks and conditions? Does missing workflow capture lower confidence rather than count against a person? Has a reviewer outside the programme challenged the interpretation?

Record finding, evidence, confidence rationale, limitation, alternative explanation, and next test. Avoid causal language unless the design supports it. Separate the observed period from later outcomes so hindsight does not rewrite the original record.

Questions about governance

Who may access detailed evidence? What content is unnecessary to retain? How are corrections recorded? When is information deleted? What happens when the instrumentation or rubric changes? Who can approve a broader use? These questions require the organization’s applicable policy and legal review.

This question set supports disciplined inquiry. It does not itself establish programme effectiveness, general capability, legal compliance, validated benchmarks, or a basis for automated employment decisions.