How to analyze user feedback with AI

Frame the decision before collecting themes

Start with the decision the team must make. “What do users think?” is too broad. A better question is: “Which obstacle should we address before expanding the assessment invitation flow to customer administrators?” Define the users, workflow stage, evidence window, and plausible actions.

Create a source ledger before asking AI to summarize anything:

| Source | Synthetic sample | What it can show | Limitation | | --- | --- | --- | --- | | Interviews | 8 administrators | Goals, language, observed confusion | Small purposive sample | | Support tickets | 43 invitation-related tickets | Recurring reported failures | Only users who contacted support | | Usage events | 610 invitation attempts | Where abandonment occurred | Does not explain intent | | Account notes | 12 implementation reviews | Organizational constraints | Uneven note quality |

The GOV.UK Service Manual advises teams to extract observations from research, organize them, determine findings, decide actions, and share results. Keep those stages separate. An AI-generated cluster is an organizing aid, not yet a finding.

Normalize evidence without erasing provenance

Give each item a stable identifier and retain its source, date, segment, task stage, and exact excerpt or event definition. AI can suggest a concise observation, but the reviewer checks it against the original. Do not merge “could not find the resend control” with “resend email arrived late” merely because both mention resending.

Use distinct labels for evidence types:

This prevents a dashboard count and an interview quotation from being treated as interchangeable proof.

• observed: a researcher saw the participant take or fail an action;
• reported: a person described an experience or preference;
• behavioral: a defined product event occurred;
• contextual: an account, policy, or implementation condition may affect interpretation;
• inferred: the team proposes an explanation that still needs testing.

Build decision themes, not word-frequency themes

Group evidence around a user decision or product mechanism. For the synthetic invitation flow, the team might find:

| Decision theme | Supporting evidence | Contradiction | Next move | | --- | --- | --- | --- | | Administrators cannot tell whether an invitation was sent | 5 observed hesitations; 18 tickets; repeated refresh events | Experienced administrators completed without delay | Prototype a visible delivery state and test with new administrators | | Resend is the main failure | 11 tickets mention resend | Events show most abandonment occurs before first send | Do not prioritize until event definition and journey are reviewed |

The second row is important. It shows how ticket volume can make a visible problem seem like the largest problem. Feedback channels have different selection effects, and importance also depends on severity, affected segment, strategic relevance, and reversibility.

Preserve disagreement and negative cases

Ask AI to find evidence that does not fit the leading theme. Reviewers then inspect those items directly. Segment by relevant context such as user experience, account configuration, device, task frequency, or accessibility need, but avoid creating tiny cuts that expose individuals or generate unstable stories.

GOV.UK guidance on planning user research recommends turning assumptions into research questions and updating research plans as teams learn. A contradictory signal can therefore become a new question rather than being discarded as noise.

Produce a traceable synthesis

The final synthesis should contain the decision question, source ledger, evidence definitions, theme table, negative cases, uncertainty, and recommended next test. Each finding points to source IDs. Each recommendation states who owns it and what evidence would change the decision.

A common failure is: “Users want a resend button because resend appeared in 26 comments.” This confuses mention frequency with importance and may double-count copied ticket language. Repair it by deduplicating items, checking journey stage and segment, comparing behavioral evidence, and testing the proposed mechanism.

Before release, verify that:

• every summarized observation can be traced to its original source;
• source types and event definitions are not blended silently;
• themes answer the stated product decision;
• contradictory and unclustered evidence was reviewed;
• volume, severity, reach, and strategic relevance are considered separately;
• the next action is a bounded test, not a claim that the interpretation is correct.

Sources and scope

This is an educational synthesis method. The synthetic example is not customer evidence and does not establish that a product change will improve an outcome.

• UK Government Service Manual, Analyse a research session: used for the progression from observations to organized findings and actions.
• UK Government Service Manual, Plan user research for your service: used for decision-focused research questions, assumption testing, and iterative planning.

Sources: [1] [2]