Enterprise AI assistant guide for AI-enabled knowledge work

Start with the work decision, not a feature list

NIST’s Generative AI Profile recommends documenting intended purposes, deployment settings, users, assumptions, limitations, data sources, and relevant test and evaluation measures. The UK National Cyber Security Centre’s secure AI development guidance treats secure design, development, deployment, and operation as connected lifecycle concerns. Together, these sources support a task-first evaluation: define the work and its risks before deciding that an assistant is suitable.

“Enterprise” does not establish what a service can access, retain, retrieve, log, or protect. Those facts can vary by product, plan, tenant configuration, integration, permission, geography, and contract. Verify them in current first-party documentation and in the organization’s own environment.

Use the ASSIST fit record

Create one ASSIST record for every material workflow under evaluation:

| Field | Decision question | Evidence to retain | | --- | --- | --- | | Aim | What bounded task and user outcome should the assistant support? | Task statement, user group, expected artifact, and excluded uses | | Sources | Which data may enter the interaction, and which sources may ground the output? | Data classes, repositories, permissions, and provenance route | | Safeguards | Which technical and procedural controls must operate? | Access control, retention, review, escalation, and incident evidence | | Inspection | What can an authorized reviewer observe without exposing unnecessary content? | Available logs, configuration records, audit events, and access rules | | Success test | What behavior and artifact will count as useful? | Scenario, rubric, error checks, and acceptance threshold | | Transition | Who owns rollout, change control, support, and exit? | Named owner, review date, fallback, and decommission route |

Complete the record with verified facts, not vendor-category assumptions. Mark each item confirmed, conditional, unavailable, or unknown. Unknown is a valid pre-pilot finding. It should trigger an evidence request, a narrower task, or a hold rather than an invented answer.

Worked example: a synthetic policy-question assistant pilot

Consider a synthetic operations team evaluating an assistant for answering staff questions about internal travel policy. This example is original and does not describe a Two Prune customer or a real deployment.

The team first writes the aim: help an employee locate the relevant policy passage and prepare a question for a human policy owner. The assistant must not approve expenses, interpret law, or replace the accountable policy owner. Approved policy documents may be used as grounding sources; employee medical details, payment-card data, and unrelated personnel records are excluded.

The ASSIST record exposes three gaps. The product documentation describes administrative controls, but the team has not yet verified the tenant’s actual retention configuration. The proposed document connector inherits repository permissions, but a test user can still retrieve a superseded policy stored in an old folder. The available reporting shows activity counts, but it does not by itself prove that an answer cited the current policy or that the user checked it.

The pilot therefore uses twenty synthetic questions across routine, ambiguous, and out-of-scope cases. Reviewers check source retrieval, quotation accuracy, current-version selection, uncertainty language, refusal or escalation behavior, and the usefulness of the employee’s final question. A version register identifies the policy in force for each case. The team records configuration evidence separately from answer-quality evidence.

The result may support a decision to proceed, narrow the use, repair the source repository, or stop. It does not establish that every future response will be correct.

Separate four evidence layers

An enterprise assistant evaluation becomes misleading when different forms of evidence are blended together:

A successful task test cannot prove that a retention setting is correct. A screenshot of a control cannot prove that users verify generated claims. Usage volume cannot prove work quality. Keep the layers distinct so the decision-maker can see which claim each item actually supports.

• Product evidence describes capabilities documented by the provider for a particular plan and date.
• Configuration evidence shows what is enabled in the organization’s tenant, including permissions, connectors, retention, and administrative controls.
• Task evidence shows how the configured service performs on representative work with known sources and expected artifacts.
• Operating evidence shows whether ownership, review, incidents, changes, and user support work after deployment.

Run a bounded evaluation sequence

Use the following sequence for a new workflow:

NIST’s AI Risk Management Framework says AI systems should be tested before deployment and regularly in operation, with limitations and deployment context documented. A one-time pilot is therefore a decision input, not permanent assurance.

• Define the task, users, stakes, prohibited uses, and expected artifact.
• Map the permitted data path from input through grounding, generation, storage, logs, and downstream export.
• Verify current product claims against first-party sources and tenant settings.
• Construct representative cases, including ambiguity, stale evidence, permission boundaries, and escalation needs.
• Test both the output and the human workflow used to check, revise, and apply it.
• Review failures by cause: source, configuration, model behavior, user action, interface, or operating process.
• Decide whether to release, narrow, repair, repeat the test, or hold.
• Schedule monitoring and re-evaluation when the model, product, connector, data, policy, or workflow changes.

What to measure without overclaiming

Useful measures depend on the workflow. They may include retrieval of the correct source version, unsupported-claim frequency, escalation quality, reviewer correction categories, time to a usable artifact, access-control test results, and unresolved evidence gaps. Record the denominator, case design, configuration, and review method so a later result is comparable.

Avoid treating prompt count, active users, token volume, or self-reported satisfaction as proof of capability. These can describe adoption or workload, but they do not independently show whether the work was framed, checked, and delivered well.

Two Prune can provide task-bounded assessment evidence from a controlled work simulation. It does not certify an assistant, establish permanent deployment safety, or make an employment decision. An organization remains responsible for product due diligence, security, privacy, legal review, configuration, and operating controls.

Sources

NIST Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile

• UK NCSC Guidelines for secure AI system development
• NIST AI Risk Management Framework Core

Sources: [1] [2] [3]