Practical guide
Investigate an implementation issue with AI
Trace a fictional failed acceptance case through expected behavior, source data and configuration before proposing a controlled correction.
Implementation consultants can use AI to organize issue evidence while preserving the difference between expected behavior, source data, configuration and customer acceptance. This fictional reporting case isolates one extra filter and tests a correction in a sandbox. It produces an issue packet, not permission to change a live environment or proof about a real product.
Freeze the expected behavior and environment
An issue cannot be reproduced until the accepted rule and tested environment are explicit.
The fictional acceptance note says the weekly report should include every active record and exclude archived records. The test occurred in a sandbox on September 1 using source extract version 4 and configuration version 2. No live customer system, actual customer data or Two Prune feature is involved.
Ask AI to create a case header with expected rule, environment, source version, configuration version and reporter statement. Keep the reporter statement as reported evidence until reproduced. Do not broaden the case to performance, permissions or unrelated report fields.
Reconstruct the source population
A missing-row claim needs a stable population and a checked denominator.
The fictional source extract contains twelve stable record identifiers: ten are active and two are archived. Under the accepted rule, the expected output is ten active records. Count the states directly and retain the two archived identifiers as a negative-test set.
Ask AI to group identifiers by supplied status, then verify the list manually. Do not deduplicate by display name or infer status from whether a row appears in the report. Source data is evidence for the expected population, not proof that configuration executed correctly.
Compare output with configuration
The first useful comparison localizes where expected and observed behavior diverge.
The sandbox report contains nine records. All nine are active, but the tenth active record belongs to the South region. Configuration version 2 includes both active equals true and region equals North. The accepted rule mentions no region filter, so the extra filter is a supported configuration mismatch in this exercise.
Ask AI to build a trace table from each acceptance condition to source records, configuration fields and output evidence. Verify the actual configuration value. Do not call the software defective when the observed result follows the supplied configuration, and do not assume who introduced the extra filter.
Test a correction without changing live state
A proposed correction should be isolated, reversible and checked against positive and negative cases.
Copy configuration version 2 to a fictional sandbox draft, remove only the region filter and retain active equals true. The rerun produces ten active records, including the South record, while the two archived records remain excluded. Record expected and observed identifiers, not only a passed label.
AI may draft the focused test steps and compare lists, but the worker should verify the counts and unchanged filter. Do not publish, deploy or overwrite configuration version 2. A sandbox result supports a correction proposal; it does not authorize a production change.
Handle a contradictory follow-up
New evidence should update the issue theory without erasing the original reproduction.
A fictional customer note then says South records should actually be excluded for one regional team. That conflicts with the accepted rule rather than disproving the reproduction. Preserve the ten-record result and route the scope question to the named customer owner in the exercise.
Ask whether the acceptance note needs a regional variant or the reporter used a different audience context. Do not choose a requirement from the newest message alone. If the owner changes the rule, create a new acceptance version and repeat both positive and negative tests against it.
Deliver an issue packet with pending acceptance
The handoff should separate a reproduced mismatch, a tested proposal and the authority still required.
Include case boundary, accepted rule, twelve-record source population, nine-record observed output, extra-filter trace, ten-record sandbox result, archived negative test, contradictory follow-up and open owner decision. Mark configuration change and customer acceptance pending.
Review identity, denominator, version and authorization discipline. The implementation-consultant parent builds readiness evidence before acceptance. The customer-operations guide reconciles service-event metrics. This task begins with a failed implementation case and isolates configuration evidence without turning a tested correction into an approved release.
Sources and scope
NIST: AI risk management. Skills England: workplace AI foundations.
These references provide background, not validation or endorsement of this exercise. The configuration, records, calculations and test results are original fictional instructional material. Use them to discuss observable work, not to infer customer outcomes, professional credentials or performance in every setting. Before adapting the exercise, confirm the relevant acceptance criteria, approved tools, data permissions and decision owners. If the source or expected rule changes, rerun the trace and tests.
Sources: [1] [2]
Sources
- 1.AI RMF Core · NIST
- 2.AI foundation skills for work benchmark · Skills England