AI-enabled software work: require a change evidence packet before merge

Define the change claim

Write one sentence that says what behavior is changing, for whom, under which conditions, and what must remain unchanged. Then identify the repository, component, interface, data path, deployment target, and users that could be affected.

Do not use “the tests pass” as the change claim. A test suite can pass while the code addresses the wrong requirement or omits an important boundary. The NIST Secure Software Development Framework publications page lists the final SSDF Version 1.1, a final community profile for generative AI and dual-use foundation models, and the later Version 1.2 draft. Their status labels matter: a draft should not be presented as settled guidance.

CISA and the FBI's updated Product Security Bad Practices guidance is voluntary guidance for software manufacturers. It emphasizes product-lifecycle security and identifies practices the authors consider especially risky. A team can use that guidance to inform its review questions without claiming that a packet demonstrates conformity.

Sources: [1] [2]

Assemble six pieces of evidence

Create the packet alongside the pull request:

Keep the packet about the change, not about proving that an AI tool is generally safe or useful. The same developer may use several tools during one task. What matters is whether the resulting change claim is supported.

• Requirement trace: issue, acceptance condition, constraint, and explicit non-goal.
• Change map: files, interfaces, data structures, permissions, dependencies, and operational surfaces touched.
• AI contribution record: what the system proposed, what the developer accepted or changed, and which assumptions required checking.
• Test matrix: expected behavior, failure paths, boundary inputs, regression surfaces, and observed results.
• Review ledger: concern, evidence inspected, disposition, owner, and unresolved uncertainty.
• Release condition: deployment sequence, monitoring signal, stop trigger, rollback route, and post-release owner.

Worked artifact: a synthetic invoice-service patch

A fictional team asks an AI coding assistant to fix duplicate reminders in an invoice service. The assistant proposes storing the last reminder date and adds a unit test. The diff is tidy and the new test passes.

The packet shows that the real requirement is narrower: prevent a second reminder within the customer's local calendar day while preserving retry behavior after a failed send. The proposed field stores a UTC date, the queue can retry after a timeout, and the service supports customers in several time zones.

The change map identifies the scheduler, invoice table, notification provider, retry queue, and audit event. The test matrix includes:

The midnight and concurrency tests fail. The developer replaces the date flag with an idempotency key derived from invoice, reminder type, and customer-local service window, then adds a transactional uniqueness check. A reviewer verifies that a provider timeout does not incorrectly suppress the allowed retry. The release condition includes a duplicate-event monitor and a rollback path that does not delete audit history.

The example is synthetic. It does not establish that this design is appropriate for a real billing system, and it is not a security assessment of any product.

• two successful attempts within one customer-local day;
• a provider timeout followed by a retry;
• a reminder just before and after local midnight;
• a daylight-saving transition;
• two workers processing the same invoice concurrently;
• an invoice without a configured time zone.

Review assumptions, not just syntax

Ask the author to list external facts the AI relied on: framework APIs, library versions, schema fields, configuration defaults, error behavior, concurrency semantics, and deployment conditions. Verify material facts against the actual code, locked dependency version, migration state, and authoritative documentation.

Select tests from the change claim rather than from the generated implementation. If the code introduces a cache, test stale data and invalidation. If it changes authorization, test denied paths and tenant boundaries. If it parses input, test malformed, oversized, missing, and ambiguous cases. If it changes data, test upgrade, rollback, partial failure, and replay.

Record a disposition of merge, revise, split, or hold. An unresolved question can be acceptable only when the owner, consequence, and follow-up condition are explicit and the release boundary allows it.

Keep evidence attached through release

After merge, link the deployed revision, migration result, monitoring window, incidents, and rollback decision back to the packet. Reopen it if the dependency, requirement, input distribution, production configuration, or observed failure mode changes.

Useful process indicators include changes without acceptance conditions, AI assumptions without checks, tests that mirror only the implementation, review comments closed without evidence, and releases without stop triggers. These are signals to inspect. They do not by themselves measure developer skill or software quality.

Two Prune can assess how a participant handles a bounded software scenario, reviews source material, uses AI, tests assumptions, and communicates a release decision. It does not inspect a customer's production code, certify software security, promise outcomes, or make employment decisions.