How to evaluate an AI research assistant

Define the research task

Name the decision, audience, scope, date cutoff, permitted sources, excluded sources, and consequence of error. Distinguish discovery, summarization, comparison, and recommendation. An assistant that helps find documents has not necessarily supported the judgment made from them.

The NIST AI RMF Core calls for documenting intended use, knowledge limits, third-party components, evaluation, human oversight, and production monitoring. Apply those categories to the exact research workflow and configuration rather than treating a vendor label as evidence.

State whether the assistant searches a public index, a connected workspace, a curated corpus, or user uploads. Test permissions and freshness for the actual account. Never assume that a platform cannot access or report a source without checking its current plan, settings, connectors, and region.

Sources: [1]

Build a claim-evidence notebook

Give each material statement a claim ID. Record the source URL or document ID, publisher, publication and access date, exact supporting passage, source type, transformation, uncertainty, conflicting evidence, reviewer, and final disposition.

Use dispositions such as supported, supported with qualification, contradicted, source unavailable, outside cutoff, and unresolved. A citation that links to a relevant page may still fail to support the statement. Open the source and verify the decisive passage.

NIST's Generative AI Profile discusses evaluation, provenance, feedback, monitoring, and risks such as confabulation. Use those themes to design tests, while recognizing that they do not certify a research product or a specific answer.

Sources: [2]

Work a synthetic market-research example

Imagine a strategy team researching whether a fictional region should be prioritized for a training pilot. The assistant returns a concise summary with three claims: the market is growing 18 percent annually, a new policy takes effect next quarter, and employers prefer short practical programmes.

The notebook finds:

The analyst rewrites the memo to separate verified facts, proposals, directional signals, and unknowns. The decision becomes “conduct three bounded discovery interviews and verify the policy status,” not “launch immediately.”

The notebook preserves the rejected wording and reason. This makes later updates possible when the consultation closes or a better market source appears.

• R-01: the 18 percent figure comes from a vendor forecast for a broader geography and a different product category. Disposition: not supported for the stated market.
• R-02: the policy source is an official consultation, not an enacted rule. Disposition: qualify as proposed and record the consultation date.
• R-03: the employer-preference claim comes from six interviews in an undated slide. Disposition: limited directional evidence, not a population estimate.
• R-04: a current official labor-market source was absent from the assistant results because the connector indexed only the team's workspace. Disposition: expand the source route and rerun discovery.

Test retrieval and synthesis failures

Include a superseded document, two sources with conflicting definitions, a paywalled abstract, a table whose units differ from the narrative, a source beyond the cutoff, an inaccessible workspace file, and a document containing instructions aimed at the assistant. Verify whether the workflow preserves source identity and resists treating document instructions as task authority.

Test quotation fidelity, numeric transcription, dates, entity names, and citation destinations. Compare the cited passage with the final statement after summarization. Check whether qualifications survive export into slides or a document.

Record what was not searched. A complete-looking answer can reflect a narrow corpus. Search coverage should be described, not implied. Qualified domain reviewers decide whether the evidence is sufficient for the actual decision.

Score workflow fit

Rate source coverage, provenance, passage support, conflict handling, freshness, permission behavior, correction workflow, export fidelity, and review effort. Keep not tested separate from unsupported.

Use fit, fit with conditions, redesign, or not suitable for this task. Repeat the test when the corpus, connector, model, account permissions, or decision changes.

Two Prune can simulate research-intensive knowledge work and retain the submitted artifact plus available workflow evidence for human interpretation. It does not certify sources, approve research conclusions, monitor production tools, or make employment decisions. Assessment findings remain task-bounded and confidence-aware.