Data Leakage

Data leakage occurs when training or validation uses information that would not be available at prediction time. It is not a minor hygiene issue; it changes the estimand of supervised learning and makes evaluation metrics optimistic.

What leakage does

Leakage gives the model an answer key, or a proxy for it. The model may look excellent in model selection while learning a production-impossible shortcut. It usually enters through one of a few recurring channels:

Leakage source	Example	Fix
Preprocessing on all data	scaler or PCA fit before the split	fit inside the training fold only
Target encoding	category replaced by the mean of $y$ over all rows	compute within cross-validation folds
Temporal leakage	a feature that uses future information	split in time order
Group leakage	the same user or patient in train and test	group-aware split
Duplicate rows	near-duplicates spread across the split	de-duplicate before splitting

The clean versus leaky estimate

The intended validation estimate averages a model’s loss over a held-out set:

$${\hat{R}}_{val}=\frac{1}{|V|}\sum _{i\in V}L(y_i,{\hat{f}}_T(x_i)),$$

where $V$ is the validation set, $|V|$ its size, $L$ the loss, and ${\hat{f}}_T$ a model fit only on the training data $T$ by a learning procedure $A$, so $\hat{f}=A(T)$. Leakage means the fitted pipeline instead depends on validation labels $y_V$ or other future information — $\hat{f}=A(T,V,y_V)$ — so the model has effectively seen what it is being tested on and the estimate is optimistic.

Worked example

This snippet compares cross-validation accuracy with clean features against accuracy after adding a target-derived leaky feature.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
 
X, y = make_classification(n_samples=240, n_features=6, n_informative=3,
                           flip_y=.2, random_state=15)
leaky = y.reshape(-1, 1) + np.random.default_rng(15).normal(0, .01, size=(len(y), 1))
X_leaky = np.c_[X, leaky]
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=15)
clean = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=cv).mean()
leak = cross_val_score(LogisticRegression(max_iter=1000), X_leaky, y, cv=cv).mean()
print("clean_cv_accuracy", round(clean, 3))
print("with_target_leak_accuracy", round(leak, 3))

Observed output:

clean_cv_accuracy 0.667
with_target_leak_accuracy 1.0

The leaked feature is a noisy copy of the label, so cross-validation becomes perfect. Real leakage is often less obvious but follows the same pattern.

Caveats

Leakage often enters through feature engineering: aggregates computed over the full dataset, encodings using target means, or text fields created after outcome review. Time-aware and group-aware splitting should be chosen before looking at model performance.

References

• scikit-learn User Guide: Common pitfalls and recommended practices
• scikit-learn User Guide: Pipelines and composite estimators

Nav

Section — Classical Machine Learning

← Model Selection Feature Engineering →