Learner resource

Leakage Checklist

Leakage happens when model training or evaluation uses information that would not be available in real use.

Column leakage

[ ] Does any feature directly contain the answer?
[ ] Does any feature contain a future outcome?
[ ] Does any feature describe an action taken after prediction time?
[ ] Does any feature use final status, final score, completion date, refund status, or later review?

Split leakage

[ ] Are records from the same real-world unit split across train and test in a way that shares information?
[ ] Does a time-based problem use future rows to predict earlier rows?
[ ] Are duplicates or near-duplicates split across train and test?

Preprocessing leakage

[ ] Are imputers fit only on training data?
[ ] Are scalers fit only on training data?
[ ] Are encoders fit only on training data?
[ ] Are feature-selection steps inside the cross-validation pipeline?

Evaluation leakage

[ ] Was the test set used to choose the model?
[ ] Was the test set used to tune thresholds?
[ ] Were many models tried and only the best test result reported?

Decision

If any item is true or uncertain, pause and repair the experiment before trusting the score.