How to evaluate an AI data-analysis tool for knowledge work

Define the analytical decision first

Write the audience, decision, input data, required output, tolerable error, and escalation path. “Analyze spreadsheets” is too broad. A testable task is: compare quarterly unit economics across four regions, explain the largest change, and produce a table whose totals can be independently reproduced.

The UK Government Data Quality Framework treats data quality as fitness for purpose. That principle matters because a tool can calculate correctly from data that is incomplete, stale, or unsuitable for the decision.

Create a reference packet with a clean dataset, a version containing known defects, a calculation specification, and expected evidence. Keep the answer key separate from the tool session.

Sources: [1]

Record the tested configuration

Create a configuration card for every run:

Do not generalize from one configuration to the whole platform. Current capabilities may vary by plan, permission, region, connector, or administrator setting. Recheck the vendor's current first-party documentation before making a product comparison.

• product and plan name
• model or analysis mode, if displayed
• date and region of the test
• connected data sources and acting identity
• enabled tools or extensions
• retention and sharing settings visible to the tester
• file size, format, and row limits encountered

Use a seven-part result ledger

For each task, record ingestion, schema interpretation, transformation, calculation, explanation, provenance, and export. The GAO AI Accountability Framework separates governance, data, performance, and monitoring. A result ledger turns those broad concerns into inspectable task evidence.

Consider a synthetic regional-margin workbook. One table stores revenue in whole currency units, another stores costs in thousands, and a late adjustment duplicates two rows. The expected output identifies the unit mismatch, removes the duplicated adjustment with a traceable rule, recalculates margin, and cites the source cells.

A run record might say:

This does not produce a universal product score. It describes one configured run on a defined task.

• Ingestion: all four sheets loaded; one hidden column was omitted from the tool preview.
• Schema: the tool inferred both monetary columns as equivalent units.
• Transformation: duplicate rows were detected only after an explicit check.
• Calculation: corrected totals matched the answer key after unit normalization.
• Explanation: the first narrative attributed the change to price, although volume drove most of it.
• Provenance: table values were linked to cells, but the narrative claim lacked a source pointer.
• Export: formulas became static values in the downloaded file.

Sources: [2]

Test failures and recovery

Add cases for missing values, mixed units, duplicated records, filtered rows, date ambiguity, formula errors, and conflicting source versions. Ask the tool to state uncertainty and request clarification where appropriate. Then introduce one correction and observe whether the revised result preserves or silently changes prior assumptions.

The NIST Information Quality Standards provide a useful review lens of utility, objectivity, and integrity. Apply it to both output and trace: can the artifact support the intended decision, can decisive claims be checked, and is the transformation history protected from unnoticed change?

Require independent reperformance for decisive calculations. A reviewer should be able to reproduce the number from the source data and documented transformation without trusting the generated explanation.

Sources: [3]

Compare tools without hiding uncertainty

Use a task-by-task comparison table. Mark supported, supported with conditions, failed, or not tested. Include the tested configuration and date in every row. Separate workflow fit from result quality: a product may produce a correct table but lack the provenance or export behavior the team needs.

Two Prune can simulate an AI-assisted analytical task and retain the submitted artifact plus available workflow evidence for human interpretation. It does not certify a vendor, establish all current platform capabilities, or replace technical, security, privacy, or procurement review. The useful conclusion is a bounded one: what this person and configured tool demonstrated on this task, with which evidence and unresolved limits.