A model graded on its own study notes always looks brilliant
Machine learning only earns trust when it works on data it has never seen. That means two disciplines, learned together today. First, you split the data: train on one part, hold out the rest, and score exactly once on that held-out part. Second - the part beginners miss - every transformation you fit (a scaler, an encoder, an imputer) must also learn from the training rows only. A scaler that has seen the test set has leaked, and the score it produces is a comforting lie.
train_test_split on Lumen - stratified so the rare 3.2% conversion rate survives in both halves - plus a scar you will never forget: you will fit a scaler the wrong way (on all the data, before the split) and watch the test error collapse from an honest 0.90 to a fantasy 62.80 in scikit-learn's own worked example. Leakage is the number-one beginner mistake; after today you will smell it coming.
Why split, and the sacred rule 6 min live
The split is simple to state and easy to get subtly wrong. Train on the training rows. Fit the model - and every transformer - on the training rows only. Then score once on the test rows, which stay sealed until the very end. And because conversion is rare, we stratify the split so both halves keep the same 3.2% positive rate; otherwise a small test set can end up with almost no buyers to grade against.
Livetest_size and random_state3 min▶
train_test_split shuffles the rows and cuts them into two frames. Two arguments matter:
test_size=0.25- keep 25% of rows sealed for the final grade. 20-30% is the usual range; too small and the score is noisy, too large and you starve training.random_state=42- fixes the shuffle so your split (and your numbers) match mine and are reproducible run to run. Change it and the exact score wobbles, which is a preview of why we cross-validate in b6.
Self-studystratify for imbalanced targets3 min▶
Lumen converts only ~3.2% of the time. A plain random split can, by bad luck, put too few buyers in the test set - so your metrics are computed on a handful of positives and swing wildly. Passing stratify=y tells scikit-learn to preserve the class proportions in both halves:
from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
Now both y_tr and y_te carry ~3.2% positives. For any imbalanced classification target, stratify by default - it costs nothing and steadies every metric that follows.
Split Lumen right, then watch a leak fake the score 22 min · everyone builds
We reload the dataset you built in b1, make a correct stratified split, then deliberately stage the leakage disaster: fit a scaler on all the data before splitting (wrong) versus on the training rows only (right). scikit-learn's own documentation puts hard numbers on it, and we will name exactly why the wrong way lies.
Reload the dataset and define X and y. Every builder session starts here. We predict converted from the feature columns.
import numpy as np, pandas as pd
df = pd.read_parquet("lumen_sessions.parquet") # built once in b1
feature_cols = ["prior_30d_spend", "new_vs_returning", "channel",
"product_category", "device", "pages_viewed", "session_hour"]
X = df[feature_cols]
y = df["converted"]
print(X.shape, "| positive rate: %.2f%%" % (100 * y.mean())) # (40000, 7) | ~3.2%
Make the split - stratified. Hold out 25%, fix the seed, and confirm the rare class survives in both halves.
from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
print("train:", X_tr.shape, "positive %.2f%%" % (100 * y_tr.mean()))
print("test :", X_te.shape, "positive %.2f%%" % (100 * y_te.mean()))
# train: (30000, 7) positive ~3.2% test: (10000, 7) positive ~3.2%
Both halves keep ~3.2% positives - that is stratify doing its job. Drop stratify=y and re-run: the test rate drifts, and with a rare class that drift is enough to bend your metrics.
The leakage disaster. Fit a scaler the WRONG way (on all data, before splitting) versus the RIGHT way (on train, then transform test). This is scikit-learn's own worked example - the numbers are theirs.
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error
# --- WRONG: scaler sees the test set before the split ---
scaler = StandardScaler().fit(X_all) # <- peeks at everything
X_all_scaled = scaler.transform(X_all)
# ...split AFTER scaling -> test-set stats have leaked into training
# --- RIGHT: fit the scaler on TRAIN only, then apply to TEST ---
scaler = StandardScaler().fit(X_train) # <- learns from train alone
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test) # test transformed, never fitted
model = Ridge().fit(X_train_s, y_train)
print("correct test MSE :", round(mean_squared_error(y_test, model.predict(X_test_s)), 2))
# scikit-learn's documented figures for this example:
# correct (fit scaler on train) -> test MSE 0.90
# wrong (forgot to transform / leaked) -> test MSE 62.80
Why does the wrong way flatter the score? When the scaler is fit on all the data, the test rows help set the mean and standard deviation used to normalise training - so the model is quietly tuned to statistics of the very rows it will be graded on. The offline number looks great because the test set was never truly unseen. In scikit-learn's example the mirror-image mistake - a transform inconsistently applied - blows the honest MSE of 0.90 up to 62.80. Either way, the lesson is one line: whatever you fit, fit on train only.
The fix, previewed. The reason leakage keeps happening is that scaling and modelling are two separate steps a human has to wire up in the right order. A Pipeline binds them so fit can only ever see the training rows - leakage becomes structurally impossible. Full treatment in b9; here is the shape.
from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression pipe = make_pipeline(StandardScaler(), LogisticRegression()) pipe.fit(X_tr_numeric, y_tr) # scaler fits on train inside the pipe pipe.score(X_te_numeric, y_te) # test is only ever transformed, never fitted
A junior analyst normalised the full feature table, then split, then reported a beautiful offline accuracy to the growth team. The model shipped and immediately underperformed - because in production the scaler never gets to see the future rows it will score, so the advantage it had trained on evaporated. The offline win was borrowed from the test set. One reordered line - fit the scaler after the split, on train only - would have shown the true, more modest number up front.
This week ◐ 35 min total
- Watch the score wobble. Re-run the split with
random_stateset to 0, 1, 7, and 99. Fit a quick model each time and note how the test score moves. That wobble is exactly why a single split can be lucky - and why b6 introduces cross-validation. - Prove stratify works. Split once with
stratify=yand once without. Printy_tr.mean()andy_te.mean()for both. Confirm stratify keeps 3.2% in both halves while plain random drifts. - Say it in one sentence. Write a single sentence explaining why you never call
.fiton the test set - transformers included. If you can teach it, you own it.
Three questions before you go 🎯 ◐ 90 seconds
1 · What is the sacred rule of the train/test split?
Anything you fit - the model and every transformer - learns from the training rows only. The test set stays sealed and is transformed and scored once, never fitted. Fitting on test (or on all data) is leakage.
2 · Why pass stratify=y when the target is imbalanced?
Stratify preserves the class proportions in both splits. With a 3.2% positive rate, a plain split can starve the test set of buyers and make every metric swing; stratify steadies it. It does not change the model itself.
3 · What does data leakage do to your test score?
Leakage makes the offline score optimistic: information from the test set (like a scaler's mean and SD fit on all the data) sneaks into training, so the model looks better offline than it will ever be in production. In scikit-learn's example the honest MSE 0.90 balloons to 62.80 when the transform is mishandled.
What this session covers
This builder session covers the train/test split and the leakage pitfall on the Lumen dataset - the discipline every later metric depends on. The Pipeline that makes leakage impossible gets its full treatment in b9.
Builder Session 2 cheat sheet · pin this
train_test_split(X, y, test_size=0.25, random_state=42, stratify=y). Train on train, score once on test.Pipeline binds prep + model so fit only sees train - leakage becomes structurally impossible. Full treatment in b9.