Four ways the future gets into your training data 8 min live
sklearn's definition is the one to memorize: "Data leakage occurs when information that would not be available at prediction time is used when building the model. This results in overly optimistic performance estimates, for example from cross-validation, and thus poorer performance when the model is used on actually novel data, for example during production." Everything below is a different route by which that happens - and only the first one is obvious.
LiveThe four families, with their tells5 min▶
- 1 · Target leakage. A column that only exists because the outcome already happened -
post_purchase_flag,refund_amount,cancellation_reason, an account status updated on conversion. Tell: one feature dominates importance and the model feels magically good. Test: "was this value knowable before the thing I am predicting?" - 2 · Preprocessing leakage. Fitting a scaler, imputer, encoder, or selector on the full dataset before splitting. The test rows quietly voted on their own transformation. Tell: nothing at all - this one is invisible without reading the code. Test: does any
fitorfit_transformtouch data the model should not have seen? - 3 · Temporal leakage. A random split on data with time order, so the model trains on next month to predict last month. Tell: offline results that never reproduce online. Test: is every training row older than every test row?
- 4 · Group leakage. The same customer, patient, or device appearing on both sides of the split. The model memorizes the entity rather than learning the pattern. Tell: the score drops hard on genuinely new entities. Test: split by group, not by row.
The one that is hardest to argue with. A churn model shipped with 0.94 AUC and collapsed in its first month. The culprit was a last_contact_type field, which the retention team filled in after deciding a customer was leaving. Nobody lied, nobody was careless with the split, and the code was clean. The column simply knew the future - and no amount of cross-validation can detect a feature that is contaminated in every fold.
Self-studyThe leakage audit, as a checklist3 min read▶
Run this before every model ships. It takes ten minutes and has saved entire quarters:
- For each feature: when is this value written? If the answer is "at or after the outcome," it is out.
- For each transformer: what data did it see during
fit? If the answer includes test rows, the score is fiction. - For the split: does time matter here? If yes, split by time. Do entities repeat? If yes, split by entity.
- For the result: is this suspiciously good? A model that beats the domain expert's intuition by a wide margin is a leakage suspect until proven otherwise.
Three rules, and the one structure that enforces them 7 min live
sklearn's guidance is unusually direct, and it fits on a sticky note. These three rules kill families 1 and 2 outright, and make 3 and 4 into deliberate choices rather than accidents.
Rule 1 (verbatim): "Always split the data into train and test subsets first, particularly before any preprocessing steps." Splitting is the first line of the notebook, not the line before model.fit.
Rule 2 (verbatim): "Never include test data when using the fit and fit_transform methods." Use fit_transform on train, transform on test - the same preprocessing, learned from one side only.
Rule 3 (verbatim): "The scikit-learn pipeline is a great way to prevent data leakage as it ensures that the appropriate method is performed on the correct data subset." Structure beats discipline: the Pipeline makes the wrong thing hard to type.
LiveThe Pipeline as a leakage-proof container5 min▶
sklearn lists three purposes for Pipeline, and the third is the one this session cares about, verbatim: "Pipelines help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors."
The Lumen version, with every transform inside the container where cross-validation can protect it:
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score
import numpy as np
X_train, X_test, y_train, y_test = train_test_split( # rule 1: FIRST
X, y, test_size=0.4, stratify=y, random_state=42)
prep = ColumnTransformer([
("num", StandardScaler(), make_column_selector(dtype_include=np.number)),
("cat", OneHotEncoder(handle_unknown="ignore"),
make_column_selector(dtype_include=object)),
])
pipe = Pipeline([ # rule 3: structure
("prep", prep),
("select", SelectKBest(f_classif, k=25)), # inside, not before
("model", LogisticRegression(max_iter=1000)),
])
pipe.fit(X_train, y_train) # rule 2: train only
print(cross_val_score(pipe, X_train, y_train, cv=5).mean())Note where SelectKBest sits. Outside the pipeline it sees every row before the split; inside it re-fits within each cross-validation fold, which is the entire difference between the two numbers in the next section.
Self-studySplitting when time or entities matter4 min read▶
Families 3 and 4 need a different splitter, not a different transformer:
- Temporal:
TimeSeriesSplit, or a hard date cutoff. Every training row must predate every test row. If you would not have had the data on that date, you do not get it. - Group:
GroupKFold/GroupShuffleSplitwith the customer or device id as the group. This is the default for anything with repeat customers - which is most commercial data, Lumen included. - Both: group-aware, time-ordered splits. Fiddly, and still cheaper than discovering the problem in production.
Target leakage, priced ★ 10 min · everyone runs it
The workbench opens with all five honest feature groups on - the 0.813 you earned across b1 to b4. Switch on target leakage, read the training score, then press Score on held-out and watch what the column is actually worth.
Training AUC climbs to 0.967. Held-out: 0.789. The gap is 0.179 - but the number that matters is the comparison to 0.813: the leaked model is worse than the honest one.
Why worse, not merely equal? Because the model spent its capacity on the leaked column and de-weighted the real features. When the leak evaporates at prediction time, what is left is a model that stopped paying attention to the signal it actually has.
That is the true cost of leakage: not a wasted feature, but a corrupted model plus a launch decision made on a number that was never real.
The leak that survives a correct split ★ 10 min · everyone runs it
Demo 1 was the loud kind. This is the quiet kind, and it is the one that catches experienced people: you split correctly, you never touch the test set, and you still leak - because the encoder learned each row's outcome and then handed it back to that same row. Below, postal_code with 1,200 levels, drawn independently of the target. It knows nothing. Watch it score anyway.
Naive encoding: train AUC 0.931. From a column that is statistically pure noise. Every row's own conversion helped compute its own encoded value, so the feature is partly a copy of the answer.
Held-out: 0.493. Chance. The signal was never there; the training score was an echo.
Cross-fitted encoding - each fold encoded using the other folds, which is exactly what TargetEncoder.fit_transform does internally: train 0.509, held-out 0.493. Honest and boring, which is the goal.
Remember the asymmetry: the naive version did not just inflate the score, it would also have won a feature-importance ranking and survived a review that only looked at the split.
sklearn documents the same failure at scale. Their target-encoder example uses 50,000 rows and a near-unique column with ~45,000 distinct values, independent of the target. With cross-fitting: train R² 0.800, test 0.793. Without it: train 0.858, test 0.634 - which their docs call "catastrophic overfitting." This is why fit(X, y).transform(X) is explicitly discouraged for TargetEncoder: only fit_transform applies the internal cross-fitting. Full treatment of the encoder itself is b3.
Try it yourself - this week ◐ 30-40 min total
- Run the leakage audit checklist against a model you or your team currently run. Write down every column whose write-time you cannot confidently state - that list is the deliverable, not a clean bill of health.
- Take one notebook and move every transform inside a Pipeline. Compare the cross-validated score before and after. If it drops, you just found leakage; if it holds, you just made it structurally impossible.
- Check your split: do entities repeat? Re-run with
GroupKFoldand compare. On repeat-customer data this often costs several points - points you never really had. - Optional read: sklearn's "Common pitfalls and recommended practices," the data-leakage section end to end. It is short, and it is the source for everything in Part 2.
Official sources covered
Certificates and graded exercises stay with the official providers (linked each session). This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A colleague fits StandardScaler on the full dataset, then splits into train and test. What has happened?
sklearn's rule 2 covers exactly this: never include test data in fit or fit_transform. It is invisible in the output and only findable by reading the code - which is why rule 3 (put it in a Pipeline) exists.
2 · The leaked model scored 0.789 on held-out, below the 0.813 it had without the leak. Why worse rather than merely equal?
This is the hidden cost. A leak does not sit harmlessly beside good features - it competes with them during fitting, and it wins, because in the training data it genuinely is the best predictor.
3 · A random postal code scored train AUC 0.931 with naive target encoding. What made an uninformative column look predictive?
Cross-fitting fixes it by encoding each fold from the other folds - which is why TargetEncoder.fit_transform does it internally and why fit().transform() on training data is explicitly discouraged in the docs.