AI tool selection for knowledge work: choose against a task contract

Define the task before comparing tools

Write down the decision or artifact, permitted data, required evidence, consequence of error, review owner, and any external action. Separate capabilities that are essential from features that are merely convenient. A general claim that a tool is powerful says little about whether it fits a particular task.

The NIST AI RMF Core calls for documenting intended purpose, context, requirements, specific tasks and methods, knowledge limits, human oversight, third-party components, testing, and monitoring. The framework is voluntary and under revision. Use these questions to structure a selection record, not to claim certification.

Sources: [1]

Use the MATRIX selection record

Create one record per tool and task using six dimensions:

For each dimension, record confirmed evidence, unknowns, an owner, and a disposition: suitable, suitable with controls, pilot only, or unsuitable for this task. Revisit the record when the model, plan, configuration, connector, permissions, policy, or work context changes.

The UK Government's Artificial Intelligence Playbook includes a principle to use the right tool for the job and discusses capabilities, limitations, risks, data security, selection, procurement, deployment, and lifecycle management. Its requirements apply to UK government work. Other organizations must use their own policies, contracts, and accountable specialists.

• Mission fit: What exact step does the tool support, and what output must it produce?
• Access and data: Which information classes enter the tool, where are they processed, and who can retrieve them?
• Testing evidence: Which representative cases, edge cases, and failure cases have been run?
• Review and authority: Who checks the output, and what can the tool read, draft, recommend, or change?
• Integration and exit: How does evidence move into and out of the tool, and what happens if the service changes or fails?
• eXposure: What is the impact of a wrong output, data disclosure, unavailable service, or unintended action?

Sources: [2]

Worked artifact: a synthetic contract-summary task

A fictional procurement-operations team needs to prepare a comparison table from twelve approved supplier agreements. The table must show renewal date, notice period, data-location term, and the exact contract clause supporting each entry. No system may send notices or amend a supplier record.

The team considers three options: a public chat assistant, an approved enterprise assistant with document retrieval, and a deterministic extraction script with manual review.

| MATRIX dimension | Public assistant | Enterprise assistant | Extraction script | | --- | --- | --- | --- | | Mission fit | Can summarize pasted text | Can retrieve approved files and cite passages | Can extract known fields from stable formats | | Access and data | Contract upload is not approved | Workspace and retention settings require verification | Runs inside the approved environment | | Testing evidence | No retained test pack | Ten synthetic agreements tested; two citation mismatches found | High extraction accuracy on standard clauses; weak on scanned amendments | | Review and authority | No approved reviewer flow | Draft-only access; analyst checks every clause | Analyst checks exceptions and all high-impact fields | | Integration and exit | Manual copy with weak traceability | Export includes source references | Output table retains file and page identifiers | | Exposure | Unapproved disclosure and unsupported claims | Citation errors and connector gaps | Missed clauses in non-standard documents |

The public assistant is marked unsuitable because the data route is not approved. The extraction script handles standard agreements, while the enterprise assistant is piloted on non-standard text with draft-only permissions and clause-level review. A failed citation or unreadable scan becomes an exception rather than a filled value.

This example is synthetic. It does not evaluate a named commercial product or provide procurement, security, or legal advice.

Test the choice with a bake-off

Use the same controlled pack, output contract, and scoring rules for each eligible tool. Include ordinary cases, conflicting clauses, missing pages, scanned text, tables, dates in different formats, and instructions embedded inside source files. Retain the source pack, configuration, output, corrections, elapsed effort, and final disposition.

Score what matters for the task: correct extraction, evidence traceability, missing-data behavior, reviewer effort, access control, and safe failure. Do not declare a winner from fluent prose or a single demonstration. A cheaper or faster tool may still be unsuitable when its errors are hard to detect or its data route is unacceptable.

Assess tool-selection judgment

A controlled assessment can provide a task brief, several tool descriptions, data constraints, and sample outputs. Observable evidence includes whether the participant defines the task, rejects unnecessary AI use, checks data and permissions, compares tools on representative cases, limits action authority, and records unknowns.

Report only behavior observed in that scenario. One exercise does not establish broad technical competence, organizational approval, future performance, or safe deployment. Two Prune provides task-bounded assessment evidence for human interpretation and does not make employment decisions.