Selecting an AI model for a knowledge-work use case

Turn the workflow into selection requirements

Describe the task, users, inputs, output, decision, consequence, languages, latency needs, accessibility, deployment boundary, data handling, integration, review, and failure response. Separate essential requirements from preferences. A strong model on a public benchmark may still be unsuitable for the local source set, interface, or control environment.

Treat the evaluated system as more than a model name. Record the exact version, provider or deployment, settings, system instructions, retrieval, tools, safety configuration, and test date. A result from one configuration should not be silently applied to another.

The NIST AI RMF Core asks organizations to define the tasks an AI system will support, document context and risks, and use repeatable testing, evaluation, verification, and validation. It is voluntary guidance and does not identify the right model for a particular buyer.

Sources: [1]

Build the use-case selection dossier

Create a requirement matrix with the acceptance condition, test case, authoritative expected evidence, result, reviewer, uncertainty, and disposition for each candidate. Cover task quality, source faithfulness, calculation, abstention, conflicting evidence, privacy and security requirements, human oversight, accessibility, operational reliability, provider dependency, and total operating effort.

Use the same representative test pack where comparison is meaningful, but permit candidate-specific configuration that reflects realistic use. Keep a blinded review for material outputs when practical. Record retries, manual corrections, exclusions, and failures rather than selecting only the best example.

Keep quality and operating evidence separate. A candidate can produce the stronger answer while requiring more review, less predictable latency, or an unavailable deployment arrangement. The dossier should expose these tradeoffs and state which requirement determined the decision.

Define who may accept residual uncertainty and for how long. A technical evaluator can report results, but the accountable process owner should decide whether the remaining risk fits the use. Record minority reviewer views and any condition attached to the decision.

The NIST Generative AI Profile describes generative-AI risks and suggested actions across governance, mapping, measurement, and management. It supports risk-informed test design but is not evidence that a candidate is suitable or that a local control works.

Sources: [2]

Worked artifact: a synthetic policy-summary comparison

Imagine a synthetic customer-operations team comparing two model configurations for summarizing English and Spanish service policies. The dossier requires correct authority, preservation of exceptions, traceable citations, clear uncertainty, and a stop when source versions conflict.

Candidate A produces fluent bilingual summaries but omits a regional exception in Spanish. Candidate B preserves the exception but cites an obsolete policy when retrieval contains two documents marked current. Both fail an essential requirement in the first round for different reasons.

The team repairs the test environment's version metadata, configures a source-conflict stop, and reruns new cases. Candidate B activates the stop and preserves citations; Candidate A still compresses one exception. The dossier selects Candidate B for restricted practice with human review and records the remaining absence of production-volume evidence.

This is a synthetic method example, not a comparison of named commercial models. It does not establish future performance, provider quality, legal suitability, or Two Prune customer outcomes.

Verify the decision over time

Reperform selected cases on the deployed configuration and compare logs, sources, outputs, corrections, and reviewer decisions with the dossier. Test degraded retrieval, ambiguous instructions, uncommon inputs, and realistic workload. Confirm that stop mechanisms work in the user interface, not only in a test harness.

Reopen selection after model, provider, terms, configuration, retrieval, data, workflow, population, consequence, or cost structure changes. Two Prune can observe participant judgment and verification in controlled simulations. It does not select or validate customer models, certify controls, or automate employment decisions.