How to Evaluate an AI Coding Assistant in Real Development Work

Define the task and environment

Choose a task that represents actual work without exposing production secrets or customer data. Record the repository revision, language and framework, available tools, dependency state, network access, assistant configuration, allowed files, prohibited actions, and acceptance tests. State whether the assistant may run commands, inspect history, or modify multiple files.

NIST's Secure Software Development Framework organizes secure development practices around preparing the organization, protecting software, producing well-secured software, and responding to vulnerabilities (NIST SSDF). NIST IR 8397 discusses how automated code generation affects secure software development practices (NIST IR 8397). These sources support the evaluation dimensions; they do not verify a specific assistant's present capabilities.

Write acceptance conditions before the attempt. Include functional behavior, tests, error handling, security properties, code-style constraints, documentation, and allowed scope. Preserve the initial prompt and meaningful iterations so reviewers can distinguish assistant output from developer intervention.

Sources: [1] [2]

Inspect the change as a software supply artifact

Review the diff, not just the final demo. Check for unnecessary dependencies, invented APIs, disabled controls, copied secrets, broad permissions, unsafe defaults, silent error handling, and edits outside scope. Run deterministic tests and static checks where appropriate. Inspect test quality because generated tests can reproduce the same misunderstanding as generated code.

Record provenance at a practical level: task instructions, assistant version when exposed, code revision, files touched, commands executed, dependencies introduced, reviewer changes, and final disposition. Do not claim that a platform cannot access or report data until its current product documentation, plan, permissions, and configuration have been checked.

Worked authentication-bug example

Imagine a synthetic service where an endpoint returns a project by identifier. The task asks the assistant to prevent users from reading projects outside their organization. The assistant adds an organization filter and a unit test showing an authorized request succeeds.

The reviewer notices that the test never creates a project in a different organization. A new negative test reveals that an administrator path bypasses the filter. The repaired change moves authorization into the shared data-access function, adds authorized and unauthorized cases, verifies error behavior, and documents the remaining admin exception for owner review.

The evidence packet contains the original requirement, repository revision, before-and-after diff, test outputs, threat note, review comments, and final owner decision. Passing the exercise shows evidence for this task under these conditions. It does not prove broad developer capability or production safety.

Compare assistants without a leaderboard illusion

If comparing tools, freeze the task set and environment, rotate task order, record failures as well as completions, and separate assistance time from total review time. Report task-level evidence and confidence limits. Different models, extensions, plans, repository indexes, and policy settings can materially change results.

Avoid one aggregate score that hides critical failures. A faster completion with an authorization defect is not automatically better than a slower verified change. Compare dimensions such as scope control, functional correctness, security review burden, test usefulness, explanation quality, and recovery after feedback.

Coding-assistant verification checklist

An AI coding assistant should be judged by the inspectable software change and review burden it produces. A polished answer or passing happy-path test is not enough.

• The repository revision and environment are reproducible.
• Permissions, network access, configuration, and tool version are recorded.
• Acceptance and security conditions precede the attempt.
• Review covers the diff, dependencies, tests, commands, and out-of-scope edits.
• Positive, negative, edge, and failure cases are exercised.
• Human corrections and unresolved risks remain in the evidence packet.
• Findings are limited to the tested task and conditions.