Questions That Reveal the Quality of AI Adoption

Ask whether the opportunity is real

Begin with the work context. Which tasks are in scope? Do people have approved tools, relevant access, enough time, and permission to use them? Does the task actually create an opportunity to demonstrate the behavior being discussed? What non-AI route remains available?

NIST’s AI RMF Core recommends documenting intended purpose, context, user expectations, risk tolerance, human oversight, and the limitations of evaluation (AI RMF Core). Its Generative AI Profile adds risk-management considerations specific to generative systems (Generative AI Profile). These sources are useful prompts for inquiry, not a certification checklist.

If opportunity differs between groups, comparison can mislead. Low usage may reflect lack of access or task relevance rather than resistance. High usage may reflect experimentation rather than durable workflow integration.

Sources: [1] [2]

Ask what people actually do

Move beyond prompt counts. How do users frame the task? Which sources do they inspect? When do they revise a model output? What triggers escalation? Which checks occur outside captured systems? Where does AI shift work to reviewers?

Pair telemetry with bounded observation, artifact review, interviews, or consistent simulations as appropriate. Treat each source according to what it can show. Tool logs may record activity but miss off-platform verification. Final artifacts may show defects but hide the path that produced them. Self-report can explain motives but is not direct observation of performance.

Ask whether outputs are fit for their purpose

Define quality for the artifact and audience. Relevant questions include:

Avoid one generic quality rubric across contracts, analysis, customer communication, code, and decision memos. Their evidence and consequence structures differ.

• Are material claims traceable to current sources?
• Are calculations reproducible?
• Are uncertainty and assumptions visible?
• Does the output meet task, audience, accessibility, and policy requirements?
• Which defects were caught before use, and by whom?
• What rework or review burden moved downstream?

Worked question-to-evidence map

Suppose a synthetic learning team asks whether a post-training cohort is using AI well in policy drafting. A useful map includes:

| Question | Evidence | Boundary | | --- | --- | --- | | Did participants have a fair opportunity? | Tool access, scenario design, supplied evidence | Access does not prove use | | Did they verify attributable claims? | Source map and revision history | Captured records may omit external checks | | Was the final policy usable? | Rubric-based artifact review | One task does not establish workplace performance | | Where did review effort go? | Reviewer corrections and interviews | Small samples do not show prevalence |

The team then asks which curriculum or workflow decision each answer might change. If an answer cannot influence a decision, it may be reporting noise.

A failed question set asks, “What is our adoption score, and are employees proficient?” It merges access, activity, capability, and interpretation. Repair it by splitting the constructs and specifying the evidence needed for each.

Ask about failure and dissent

Which outputs were rejected, corrected, or escalated? Which teams stopped using a tool, and why? Where do users and reviewers disagree about acceptable quality? What evidence would falsify the current success story? What unintended effects, including workload, privacy, accessibility, or overreliance, need monitoring?

Questions about failure should support learning, not automatic personnel judgments. Aggregate trends and task-bounded evidence need appropriate access, privacy, and human interpretation.

Review the question set

Good questions make uncertainty actionable. They do not turn adoption activity into a claim of universal capability or business impact.

• Every question names a construct and a decision it can inform.
• Evidence sources match the construct rather than serving as convenient proxies.
• Denominators, windows, samples, and missingness are explicit.
• Opportunity, behavior, artifact, and outcome questions are all represented.
• Dissenting and negative evidence can be recorded.
• Privacy and governance constraints are built into collection.
• Conclusions remain bounded to the observed task and population.