How to investigate an anomaly with AI

Write an anomaly statement

Describe the metric, observed value, expected baseline, time window, affected segments, data source, and detection method. Avoid causal language.

For example: “Completed assessment submissions were 18% below the trailing eight-week weekday baseline between 10:00 and 16:00 UTC on Tuesday; the decline appears in two browser segments and the warehouse table was refreshed at 17:00.” This synthetic statement identifies what moved without saying why.

Freeze the metric definition

Record numerator, denominator, inclusion rules, event name, timezone, late-arrival policy, deduplication rule, and query version. Compare the current definition with the one used for the baseline. A dashboard label is not a sufficient definition.

NIST’s statistical handbook treats time-order plots, process behavior, and exploratory analysis as tools for identifying structure and unusual observations. The practical lesson is to examine the series in context rather than judging a single point. The handbook is a statistical reference, not proof of the cause in this example.

Use a three-lane investigation

### 1. Measurement lane

Check event emission, schema changes, dropped or duplicated records, clock and timezone changes, pipeline jobs, join behavior, backfills, and dashboard filters. Compare an independent source where possible—for example, application records versus analytics events.

### 2. Comparison lane

Compare the same weekday and hour, relevant seasonal periods, unaffected metrics, regions, device or browser groups, product versions, and control cohorts. Look for a break that begins exactly at a deployment or pipeline boundary.

### 3. Behavior lane

Only after measurement checks, investigate changes in traffic mix, workflow friction, customer activity, communications, outages, or external events. Label each explanation as a hypothesis until evidence connects it to the change.

Worked investigation matrix

| Test | Result | Interpretation | Next step | | --- | --- | --- | --- | | Application submissions vs analytics completion event | Application records flat; analytics event down | Measurement issue likely | Inspect client event delivery | | Browser segmentation | Decline concentrated in browser B after version release | Narrows fault domain | Reproduce on that version | | Warehouse freshness | On time | Warehouse delay less likely | Keep as cleared check | | Start-event volume | Stable | Demand drop less likely | Examine completion instrumentation |

The evidence supports “analytics undercount is likely,” not “browser B caused lost submissions.” Reproduction and raw-event inspection are still required.

Ask AI to challenge the leading hypothesis

Give the model the metric contract and test results, then request the strongest alternative explanation, evidence that would falsify the lead, and the cheapest discriminating test. Do not ask it to infer root cause from a chart alone. Redact sensitive data and use approved tools.

Produce an anomaly brief

Include the neutral anomaly statement, metric lineage, timeline, comparison plots, hypotheses ranked by current evidence, tests completed, unresolved limitations, immediate containment, and next owner. Keep “detected,” “correlated,” “reproduced,” and “confirmed” as distinct states.

Verification checklist

Confirm the baseline uses the same definition; inspect raw records; check freshness and revisions; compare at least one independent measure; account for calendar effects; retain failed hypotheses; reproduce the issue where possible; and document when the result changed from correlation to confirmed mechanism.

Sources and claim map

All values, systems, and results are synthetic. This resource does not claim that AI can establish causality or assure detection accuracy.

• NIST/SEMATECH, e-Handbook of Statistical Methods: supports using exploratory analysis, time-order information, and statistical process methods to investigate data behavior.
• NIST/SEMATECH, Time series plots: supports examining location, variation, outliers, and patterns over time.

Sources: [1] [2]