AI fluency for data scientists: inspect evaluation leakage

Define the prediction moment

A feature is usable only if its information belongs to the intended decision context.

The fictional task predicts whether a ticket will take more than a day to resolve, using information available when the ticket is opened. The dataset includes opening category, initial description, final resolution code and total agent messages. The last two fields are accumulated during or after the work.

Ask AI to propose candidate features, then examine when each would be available. Final resolution code cannot be treated as an opening-time input merely because it appears in the historical dataset. The task definition, not column availability, determines the boundary.

Separate feature availability from relevance

A field may be strongly related to the outcome and still be unsuitable.

Total agent messages could correlate with resolution duration in the fictional history, but the final count is not available at opening. Using it would answer a different question. If a later prediction moment is desired, define that new task explicitly rather than silently changing the interpretation.

The scikit-learn documentation discusses leakage and the need to keep test information out of model fitting. That source supports the general caution; the ticket fields and exercise here are invented. Inspect generated code and reasoning for violations of the chosen information boundary.

Sources: [1]

Design a small audit before trusting a score

Start with checks that could invalidate the evaluation even if the metric looks strong.

Create a feature ledger containing the field, meaning, availability time and inclusion decision. For this exercise, retain opening category and initial description as candidates, exclude final resolution code and final message count, and record why. This is a task-design decision, not evidence that the retained features will perform well.

Check any transformation that learns from data, including preprocessing choices. Ask whether it used information from the evaluation set. Do not assume that splitting rows late in a notebook repairs earlier steps that already learned from the full dataset.

Choose a split aligned with the intended use

The evaluation design should match the question being asked about future application.

If the intended use concerns later tickets, consider how a time-based separation would reflect that setting. The exercise supplies no dates or distribution evidence sufficient to select a final protocol, so write the required checks rather than inventing a performance result.

Ask AI to propose a plan and list its assumptions. Inspect whether repeated or related records could cross the chosen boundary. Keep alternative evaluation designs visible when they answer different questions. No single split method is presented here as correct for every dataset.

Report the audit before the headline metric

A result should disclose whether the test design supports its interpretation.

Deliver the prediction definition, feature ledger, preprocessing questions and proposed evaluation boundary. If the original experiment used unavailable features, label its score unsuitable for the opening-time claim. Do not adjust the score numerically to pretend the design flaw has been corrected.

Review problem definition and leakage detection within this fictional exercise. The BI analyst guide examines dashboard comparisons; this guide concerns whether a predictive evaluation uses information that would actually be available when the model is supposed to help.

Sources and scope

The references below are background reading, not evidence that this fictional exercise has been validated or endorsed.

The case details, calculations and suggested review questions are original instructional material. Use them to discuss observable work, not to infer customer outcomes, professional credentials or performance in every setting. Before adapting the exercise, confirm the relevant facts, approved tools, data permissions and decision owners. If you change the case, revisit the expected answers and checks as well. These examples describe practice tasks, not a promise that a particular product includes the fictional features.

Sources: [1] [2]