How to investigate a quality issue with AI

Define the defect without explaining it

Write a neutral problem statement: what failed, where, when, how often, and against which specification. Do not include a cause in the statement. Preserve samples, timestamps, machine or system records, process versions, and operator notes with their provenance.

Create an evidence inventory:

| Evidence | Provenance | What it can show | Limitation | |---|---|---|---| | Defect sample | Batch 24-17, retained item | Physical failure pattern | One item may not represent the batch | | Process log | Line export, 14:00–16:00 | Recorded settings and alarms | Does not prove the sensor was accurate | | Operator note | Contemporaneous shift note | Observed intervention | Memory and terminology may vary | | Comparison sample | Prior conforming batch | Difference in output | Other inputs may also differ |

Ask AI to extract candidate facts into this structure and link each statement back to its record. Human reviewers should resolve ambiguous terminology and access restrictions.

Map the process and competing hypotheses

Draw the process from input receipt through each transformation, inspection, handoff, and release. NIST describes process mapping as a way to visualize steps and identify where evidence is generated. Its process-model guidance also emphasizes understanding inputs, outputs, controls, and variation. These references support disciplined mapping; they do not establish a cause in a particular investigation.

For a synthetic case, imagine exported assessment reports contain broken evidence links after a template update. Candidate hypotheses might include malformed source identifiers, a migration mapping error, stale cached templates, or an export renderer defect.

| Hypothesis | Expected observation | Test | Result needed to support | |---|---|---|---| | Source IDs malformed | Same records fail in every renderer | Validate IDs against schema | Invalid IDs align with failures | | Migration mapping error | Only migrated records fail | Compare migrated and new cohorts | Failure follows migration status | | Stale template | Failure follows template version | Re-render identical record by version | Defect appears only on old version | | Renderer defect | Valid records fail in one export path | Compare HTML and PDF outputs | Failure isolated to renderer |

This table prevents the first plausible explanation from becoming the conclusion.

Establish and challenge causal evidence

Run controlled comparisons where possible. Change one factor, reproduce the defect, remove the factor, and see whether the defect disappears. When control is impossible, state the strength and limits of the evidence. The US Department of Energy root-cause guidance describes systematic investigation and causal-factor analysis in its domain; use it as a method reference, not a claim that every quality problem has a single cause.

A weak report says, “The template update caused the broken links because both happened on Tuesday.” A stronger report states that the defect was reproduced only with template version 7, remained absent with version 6 using the same record, and disappeared after correcting a version-specific mapping—while noting the sample tested.

Verify correction and prevention

Separate containment, correction, corrective action, and effectiveness evidence. Restoring affected reports is containment. Fixing the mapping is correction. Adding version-contract tests may address recurrence. An effectiveness check tests a defined later sample after deployment.

Before closing the report, confirm:

The evidence artifact is the root-cause report with its inventory, process map, hypothesis register, test results, and effectiveness record.

• the problem statement contains no assumed cause;
• original records and samples remain traceable;
• the process map reflects actual execution;
• alternative hypotheses received fair tests;
• causal language matches the evidence strength;
• containment and corrective action are distinct;
• owners and due dates exist; and
• the effectiveness check has a sample, period, and acceptance rule.

Sources

DOE Root Cause Analysis Guidance Document

• NIST process mapping
• NIST process models

Sources: [1] [2] [3]