Finding Calculation Errors in AI-Assisted Work

Freeze the calculation contract

Before checking a result, record the decision, formula, input definitions, reporting period, units, currency, population, exclusions, rounding rule, and expected tolerance. Save the input version. Separate mathematical correctness from whether the formula answers the intended question.

The GAO data-reliability guide evaluates data through accuracy, completeness, and applicability for the intended purpose (GAO data reliability). These dimensions matter before arithmetic begins. A correctly summed table can still be misleading when records are missing or the population does not match the claim.

Sources: [1]

Build a calculation test ledger

Create one row per check: calculation ID, risk, controlled inputs, independent method, expected result, tolerance, observed result, difference, disposition, reviewer, and evidence link. Include boundary cases such as zero, negative values, leap dates, duplicate identifiers, missing categories, mixed currencies, and values just above or below a threshold.

For financial audit evidence, PCAOB AS 1105 describes recalculation as checking mathematical accuracy and reperformance as independently executing procedures or controls (PCAOB AS 1105). This page borrows those verification concepts as educational practice; it does not say the resulting ledger is sufficient audit evidence.

Sources: [2]

Worked example: conversion-rate error

Imagine a synthetic campaign report. An AI assistant states that conversion rose from 2.4 percent to 3.0 percent, “a 25 percentage-point increase.” The arithmetic confuses relative percentage change with percentage-point change.

The calculation contract defines old rate, new rate, and two distinct outputs. Percentage-point change is 3.0 minus 2.4, or 0.6 points. Relative percentage change is the difference divided by the old rate, or 25 percent. The test ledger includes a plain-language label for each output and verifies the underlying numerator and denominator before recomputing either measure.

A second check finds that the new period excluded mobile traffic because of a tracking change. The arithmetic is correct for the captured records but the periods are not comparable. The final artifact reports the instrumentation break and withholds a trend conclusion. This demonstrates why data fitness and calculation verification are separate gates.

Use multiple independent checks

Recalculate high-risk figures in a separate worksheet or tool. Reconcile subtotals to totals and opening balances to closing balances. Trace a sample from source to output and another from output back to source. Compare expected and observed row counts. Inspect formulas for hard-coded values. Test whether changing one input produces the expected directional change. Use dimensional analysis to catch mismatched units.

When an AI system generated code, review the transformation and run known-answer fixtures. A passing fixture supports that exact case, not every possible input. Preserve failed cases and the corrected version.

Decide based on consequence and uncertainty

Set tighter checks for calculations that affect external reporting, money movement, safety, rights, or irreversible decisions. Define who may accept a difference and what evidence is required. Do not let a tolerance hide a systematic bias or a unit mistake. Escalate when source definitions cannot be reconciled.

Release record

Store the calculation contract, input hash or version, independent check, test ledger, exceptions, corrected output, reviewer, and approval. State which figures were tested and which were not. If inputs change, invalidate affected checks and rerun them.

Learn from the error pattern

Classify confirmed errors by origin: source defect, extraction, join, filter, formula, unit, date boundary, rounding, assumption, label, or narrative translation. Record whether the check detected the problem before release and which control should change. A recurring sign error calls for a different response from an isolated transcription mistake. Preserve a small regression set of de-identified or synthetic known-answer cases and run it when the calculation logic changes. The regression set supports those specified cases only; expand it when a new failure mode is found rather than assuming yesterday’s tests cover tomorrow’s inputs.

This method reduces avoidable ambiguity in calculation review. It does not establish that every value is accurate, that source data are fit, or that a deliverable meets accounting, audit, regulatory, or legal requirements.