Where we are
Session a1 made the case that features carry the return: 0.500 to 0.813 on Lumen's checkout data with the algorithm never changing. Tonight is the other half of that argument, and it is the half that keeps people awake. Because the same lever that lets a person move a score also lets a person move a score dishonestly, by accident, while doing careful work. That is leakage, and it is the reason this course spends a full session on it in both tracks.
The failure that looks like success 8 min live
Start with the definition from scikit-learn's own documentation, because it is precise and you can quote it verbatim: "Data leakage occurs when information that would not be available at prediction time is used when building the model. This results in overly optimistic performance estimates... and thus poorer performance when the model is used on actually novel data."
Strip out the machine-learning vocabulary and what is left is an old idea: the model was tested on an exam it had already seen. It scores brilliantly. It has learned nothing that transfers. And nowhere in that process does anything look wrong.
LiveThe four families, in business language5 min▶
You do not need to detect these yourself. You need to recognise the shape when a team describes their work, so you know which question to ask next:
- A column written after the outcome. The most common and the most human. A field gets filled in by an operational process once the thing you are predicting has already happened. In the historical table it is populated. At prediction time it is empty. The model leans on it entirely.
- A transform fitted on data you should not have seen. Averages, scalings, category groupings, feature selection - any of these computed across the whole dataset before splitting it quietly carries information from the test rows into the training process. The team did preprocessing "first" because that felt tidy.
- A split that ignores time. If your rows are ordered in time and the split is random, the model trains on next month and is tested on last month. It knows the future. Production never will.
- The same customer on both sides. One entity with several rows, split randomly, so the model memorises the person rather than learning the pattern. It scores well on customers it has met and poorly on the new ones you actually want to serve.
Four different mechanisms, one identical symptom: a number that is better than it should be.
LiveWhy it survives every gate you have built4 min▶
Here is the insight worth taking away from the whole session. Leakage is the only failure mode in your pipeline that shows up as a better number. Sit with what that does to a review process:
- Code review passes. The code is clean, well tested, and correct - it faithfully does the wrong thing. There is no bug to find.
- Demo day passes, enthusiastically. A high score is exactly what everyone came to see. Nobody in that room is incentivised to interrogate a good result, and asking "is this too good?" sounds like scepticism about a colleague's work.
- The launch gate passes. Whatever threshold you set, the leaked model clears it more comfortably than the honest one. Your governance process is, without meaning to, selecting for leakage.
- Real requests fail. The first gate not staffed by people who want the project to succeed. By then it is in production, it is in a slide deck, and someone has forecast revenue against it.
Evidence you can quote 6 min live
Arguments about leakage go badly when they are arguments about judgement. They go well when they are arguments about two documented results. Memorise these; they do more work than any amount of general caution.
| Evidence | Leaked result | Honest result | What it proves |
|---|---|---|---|
| sklearn's own experiment, on pure noise | 0.76 accuracy | 0.50 accuracy (0.43 under cross-validation) | 26 points of accuracy manufactured out of data containing no signal whatsoever |
| The Lumen trap, on our own case | 0.967 training AUC 0.789 held out | 0.813 held out | The leak did not merely fake a good model - it produced a worse one than never adding the column |
LiveNumber one: 0.76 from nothing at all4 min▶
scikit-learn's documentation includes a worked example that is worth describing precisely, because its power comes from the setup. They generate a dataset of completely random noise - no relationship between the inputs and the outcome, none, by construction. There is nothing to learn. The only honest score is a coin flip.
Then they do one ordinary-looking thing: they run feature selection across the whole dataset before splitting it into train and test. Everything else is textbook.
- Selection fitted on all the data, then split: 0.76 accuracy.
- Split first, selection fitted on training rows only: 0.50.
- The same honest pipeline under cross-validation: 0.43.
Twenty-six points of accuracy, conjured from randomness, by a single step done in the wrong order. Nobody cheated. Nobody was careless. The order of two operations was reversed.
Why this is the number to quote: it removes the argument that "our data is good, so we would notice." The data in this experiment had no signal at all and the score still looked respectable. A good number cannot tell you whether it earned itself.
LiveNumber two: the leak that made the model worse4 min▶
The second number is from our own case, and it is the more useful one in a room full of people who have heard general warnings before.
Lumen's honest model, with all five feature groups on, scores 0.813 held out. Add one leaked column - a flag recorded after the sale - and:
- Training AUC: 0.967. The best number anyone on the project has ever seen. This is the moment a team messages the group chat.
- Held-out AUC: 0.789. A gap of 0.179, and, more importantly, below the 0.813 they already had.
Read that last point carefully, because it defeats the most common rationalisation. The comfortable story is "the leak flattered us, so the real performance is what it always was." It is not. The model became worse than the version without the leaked column, because it learned to depend on a feature that arrives empty in production and effectively ignored the honest signals it could have used.
So leakage is not a measurement error you can subtract out afterwards. It is a design error baked into what the model learned. The cost is not "we over-reported by a few points." The cost is a shipped model that underperforms the one you would have had by doing nothing clever at all.
The churn model that predicted the retention team. A field called last_contact_type sits in the customer table and is a superb predictor of churn. It should be. The retention team fills it in after deciding a customer is leaving. The code was clean, the split was correct, the reviewer was competent, and the model was fiction: it had learned to predict what the retention team had already concluded. In production, on a customer nobody has flagged yet, the field is empty and the model has nothing to say. Nothing in a code review catches this. Only someone who knows when that column gets written does.
Press the trap yourself ★ 12 min · everyone runs it
The workbench below arrives with the five honest feature groups already switched on and the leakage lever off. It trains a real model in your browser and scores it on 1,200 Lumen sessions it has never seen. What follows is thirty seconds you will remember for the rest of the track.
First, establish the honest baseline. Press Score on held-out and read 0.813. This is a genuinely useful model on a 3.2% conversion problem. Note it down; everything after this is measured against it.
Now switch on the dashed Target leakage lever. Do not press the held-out button yet. Read the training AUC first: 0.967. Sit in that moment for a second - this is exactly what a team sees on the afternoon they think they have solved it.
Now press Score on held-out: 0.789. A gap of 0.179, and below the 0.813 you had two minutes ago. The leak did not just flatter the model. It made it worse.
Understand the mechanism, because it is the whole lesson. The leaked column is written after the sale. In the historical table it is filled in, so the model learns to rely on it. At prediction time - the only moment that matters - it is empty. The workbench scores held-out rows the way production would, so the column arrives blank and the model's strongest feature evaporates.
Switch the lever back off and confirm you return to 0.813. The honest model was always there. Somebody just had to not ship the other one.
Say the rule out loud: a feature is only a feature if it exists before the thing you are predicting. That single sentence catches the most common family of leaks in most organisations.
Four questions to ask before any model ships ◐ 5 min read
You cannot inspect a pipeline and you should not try. What you can do is ask four questions that a healthy team answers in two minutes and an unhealthy one cannot answer at all. Use them as a standing item at the launch gate, not as an interrogation after something breaks.
1 · "When is each feature written, relative to the outcome?" Not where it comes from - when. Any column populated by a process that runs after the event you are predicting is a leak, no matter how clean the code around it is. This one question catches the churn-model family.
2 · "What data did each transform get fitted on?" Averages, scalings, category groupings, feature selection. The honest answer is "training rows only." If the answer involves the whole dataset, that is sklearn's 0.76-on-noise scenario, exactly.
3 · "Does time or entity matter in how we split?" If rows are ordered in time, the split must respect it. If one customer has many rows, they belong on one side only. A random split is the default, and the default is wrong more often than people expect.
4 · "Is this result suspiciously good, and what did we check?" The cultural one, and the most valuable. A model that dramatically outperforms the previous one is a hypothesis, not a result. Make asking this normal so it costs nobody anything socially.
Self-studyWhat the good answers sound like3 min read▶
You are listening for specificity and for speed. Compare:
- Weak: "We were careful about leakage." Strong: "We listed every feature with its write-time; two were populated post-outcome and we dropped them."
- Weak: "We used a standard pipeline." Strong: "Every transform is fitted on training rows only, and the structure enforces that rather than relying on us remembering." (Your practitioner colleagues learn exactly this in b5.)
- Weak: "We did a normal split." Strong: "Split by customer and by date, because both matter here, and here is why."
- Weak: "The score is 0.95." Strong: "The score is 0.95, which is far above our previous model, so we checked X and Y before believing it."
None of this requires you to understand the implementation. It requires the team to have thought about write-time, fit-scope, split design, and their own scepticism - and to be able to say so without preparing.
Self-studyWhere the rest of the answer lives3 min read▶
Two structural defences get covered later on this track, and both are worth knowing exist:
- Structure beats vigilance. scikit-learn's own guidance is that pipelines exist to prevent this class of error, "by ensuring that the same samples are used to train the transformers and predictors." The point for you is that leakage prevention should be a property of how the work is built, not a thing people remember on a good day. Your team should not be relying on care alone.
- Some leaks are subtle enough to survive a correct split. Encoding a high-cardinality column such as postal code using outcome information is the classic case - on our own data, a column that knows literally nothing about the target scored 0.931 in training and 0.493 held out. The fix is a technique your team should be able to name; it is covered in practitioner session b3. You do not need the mechanism, only the awareness that "we split correctly" is not by itself a complete answer.
- Consistency between training and serving is the governance version of this whole session, and it is a4: feature stores, point-in-time correctness, and why "train like you serve" is a leadership concern rather than an engineering detail.
Two things to do this week ◐ 20 min total
- Take one model in production and ask its owner question one: when is each feature written, relative to the outcome? Ask for the list, not the reassurance. Note how long it takes.
- Look at your launch checklist. If it has a minimum score but no question about whether the score is believable, add the fourth question to it. That edit costs one line and catches the failure mode this session is about.
Official sources covered
Every number on this page is either from scikit-learn's own documentation or computed live in your browser on the Lumen case. Nothing here is estimated or illustrative. Certificates and graded exercises stay with the official providers, linked each session. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Why does leakage get through code review, the demo, and the launch gate?
The code is clean, the demo is impressive, and the model clears whatever threshold you set more comfortably than the honest one. Your governance process ends up selecting for leakage without anyone intending it. The first gate that catches it is real customer requests.
2 · In scikit-learn's own experiment on data made of pure random noise, what did the leaked pipeline score?
Twenty-six points of accuracy manufactured out of data with no signal in it at all, purely because feature selection was run before the train/test split instead of after. This is the number to quote when someone argues their data is too good to fool them.
3 · The Lumen leak produced training AUC 0.967 and held-out 0.789, against an honest 0.813. Why was the leaked model worse, not just falsely good?
This is why leakage cannot be corrected by subtracting a few points afterwards. It changes what the model learned. Shipping it means shipping something that performs below the version you would have had if nobody had added the column at all.