How to Evaluate an AI Agent Platform for Knowledge Work

Define the task and action boundary

Write the business task, users, inputs, output, permitted tools, data classes, destinations, approval points, non-AI alternative, and consequence of error. Distinguish an agent that recommends an action from one that executes it. Specify which actions are reversible and which require a human decision.

The NIST AI RMF Core recommends documenting intended purpose, context, knowledge limits, human oversight, third-party dependencies, testing, and lifecycle risk management (AI RMF Core). The NIST Generative AI Profile adds cross-sectoral risks and suggested actions for generative systems (Generative AI Profile). These voluntary resources do not certify any platform.

Sources: [1] [2]

Build a capability evidence matrix

Test identity, role enforcement, tool permission, source retrieval, memory scope, instruction handling, action confirmation, logging, exception behavior, rollback, and administration. For every capability, record product version, plan, configuration, test input, expected result, actual result, and evidence.

Do not state that a platform can or cannot access prompts, files, connectors, or logs without verifying its current documentation and deployed permissions. The same product may behave differently across tenants.

Worked evaluation case

Imagine a synthetic agent that prepares supplier-review packets. It may read approved records, summarize evidence, and draft questions. It may not send messages, alter supplier status, or open records outside the assigned project.

The test set includes a permitted record, cross-project record, revoked connector, conflicting instruction in a document, missing source, and attempted send. The agent correctly summarizes the permitted record but follows an embedded document instruction to ignore the assigned scope. The evaluation therefore fails the instruction-boundary case even though the final summary is fluent.

The repair may involve tool allowlists, instruction separation, record-level authorization, and confirmation. The team retests the exact path. It does not infer broad platform safety from one corrected case.

Assess operation and change

Inspect incident routing, model updates, connector changes, administrator access, retention, export, and deletion. Ask who can modify prompts, policies, tools, and memory. Define monitoring and a safe stop for unexpected behavior.

Evaluate usability too. If people cannot understand what the agent did, find its sources, or challenge an action, formal controls may not support meaningful oversight.

Compare platforms with scenario evidence

Use the same task contract and core test cases across shortlisted configurations, then record where necessary differences remain. Compare task completion, unsupported actions, source traceability, permission enforcement, recovery, review burden, administration, and total operating constraints. Do not use a single synthetic benchmark as a universal product ranking.

Include the setup effort and controls required to obtain each result. A platform that passes only after broad permissions is not equivalent to one that passes within the intended scope. Preserve failed cases and retest after configuration changes. Commercial terms, data handling, and support commitments should come from current first-party documents and qualified review.

Platform evaluation checklist

The result is a configuration-specific evidence pack, not a universal ranking or assurance claim.

Set an expiry date for the comparison. Re-run affected scenarios when the model, connector, permission, retention setting, administrator control, or commercial plan changes materially, and preserve the earlier configuration record.

• Task, data, tools, actions, and prohibited routes are explicit.
• Current version, plan, permissions, and configuration are recorded.
• Tests cover allowed, denied, missing, conflicting, and revoked cases.
• Sources, tool calls, approvals, and outcomes are reviewable.
• High-consequence actions require an appropriate human gate.
• Recovery, rollback, incident, and change processes are tested.
• Conclusions stay limited to the tested configuration and conditions.