Where tonight sits
Session 1 laid out three links carrying a score to a decision, and the second of them was an untouched draw: the test rows did not help choose the model, and nothing about the answer was already sitting in the inputs. Session 4 asked what a published benchmark number licenses. Tonight takes the second link and the published literature at the same time, because leakage is that link failing in the harshest possible way and it has been found at a scale nobody expected. Session 6 closes the track with the question that makes all of this operational: what would change your mind, written down before the result arrives.
★ What you walk out with tonight. Three numbers you can cite from memory: 17 fields, 294 papers, eight types. A reading of those numbers that is more useful than outrage, namely that a failure distributed that widely is structural rather than personal. One question that costs a sentence and filters most of the damage: what was the split, and could anything downstream of the outcome have been in the inputs. And a line drawn carefully between a shortcut and a leak, because they look similar in a review and they need entirely different fixes.
LiveNearly three hundred papers is not a story about carelessness4 min▶
The first instinct on hearing this number is to reach for a judgement about the authors. That instinct is worth putting down immediately, and not out of politeness. It is simply the wrong model of what happened. When 294 papers across 17 disciplines fail the same way, the explanation cannot be that nearly three hundred research teams were independently sloppy. Something in the shared conditions permitted it: the incentives, the review process and the tooling all allowed a result to be published without anyone catching the flaw.
That reframing changes what you do next. If the cause is individual carelessness, the fix is vigilance and better people, which is a fix nobody can actually implement. If the cause is structural, the fix is a checklist and a default of suspicion toward results you did not produce yourself. Checklists are cheap, they survive staff changes, and they work on tired people at the end of a quarter, which is when most evidence actually gets approved.
Thornbury's clinical team wants to cite three published studies in the business case for the triage model. Under the old reading, someone senior reads the papers and forms an impression of quality. Under tonight's reading, someone asks all three the same two questions about the split and the timing of the inputs, writes the answers in the business case, and marks the ones that cannot be answered. The second version takes less time and produces a document that can be argued with.
Self-studyThe words tonight keeps using4 min▶
- Leakage. Information about the answer reaching the model's inputs when it should not have. The held-out score then measures something other than the ability to predict.
- Shortcut. A feature that predicts the outcome without causing it, and that was genuinely available before the outcome. A different failure with a different fix, and the subject of part 2.
- The prediction moment. The point in time when the decision has to be made. Every input has to have existed before it.
- A structural failure. One that arises from shared conditions rather than individual error, and which is therefore predicted by the conditions rather than by the person.
- Evidence you did not produce. Any result whose data, split and pipeline you cannot inspect. Most of the evidence in any business case.
- Base rate. How often a flaw occurs in a population of results. Once it is known to be high, suspicion stops being rude and becomes arithmetic.
- Falsifier. The observation that would make you stop believing the claim, written down in advance. Session 6.
Seventeen fields, and the same flaw in each
10 min liveThe survey by Kapoor and Narayanan, "Leakage and the reproducibility crisis in machine-learning-based science", is worth knowing by its three numbers. Leakage in 17 fields. Collectively affecting 294 papers. Organised into a taxonomy of eight types, spanning everything from textbook errors through to open research problems. That last range is the part that gets skipped, and it is the part that tells you the eight are not all the same kind of thing: some are mistakes anyone can be taught to avoid, and some are genuinely unsolved.
LiveWhat a field-wide error does to a body of evidence4 min▶
A single flawed paper is a local problem: it is wrong, somebody eventually notices, and the field moves on. A flaw distributed across a literature is a different object entirely, because the usual repair mechanism stops working. Papers cite each other, reviews aggregate them, and a systematic review of studies that share a flaw inherits the flaw and gives it a more authoritative shape. The error does not cancel out with volume. It concentrates.
For a leader the consequence is unpleasant and specific: the strength of a claim in your domain is no longer well estimated by how many papers support it. A count of supporting studies is a weak signal when the studies share a method, share a tooling stack and share a review culture. What raises confidence is variety in how the result was obtained, not volume in how often it was reported, and that is a harder thing to ask for in a business case.
LiveWhy the review process did not catch it4 min▶
Peer review is good at catching claims that do not follow from the reported evidence. It is much weaker at catching a flaw in how the evidence was produced, because that part is usually described in a paragraph rather than shown. A reviewer reads "we used a held-out test set" and has nothing to check it against. The result reproduces, because rerunning the same pipeline on the same data reproduces the same flaw perfectly.
This is why reproducibility and correctness come apart here, and it is a distinction worth having in your vocabulary before somebody uses the first word to reassure you about the second. A leaked result is generally fully reproducible. Everyone who runs it gets the same excellent number. The number simply is not an estimate of what the model can do on a patient whose outcome is not yet known.
The distinction to keep is between reproducible, meaning the same inputs give the same outputs, and valid, meaning the number is an estimate of the quantity claimed. Session 1's link two is where they part company. A result can be reproducible to six decimal places and still be an estimate of nothing you can use.
Self-studyHow to talk about this without indicting anyone3 min▶
There is a version of this session that turns into a morality tale, and it makes teams defensive and unhelpful. The better framing to use inside your own organisation is that the base rate for this flaw is high, so checking is now routine rather than accusatory. You check a wire transfer against a second pair of eyes without implying the finance team is dishonest. Same move, same tone.
Worth saying out loud in the room: your own team's results are in the same population as everybody else's. A leader who applies the checklist only to external evidence has built a filter with a hole in the middle of it, and the hole is exactly where the work you are most invested in sits.
Thornbury's ward was not leakage, and the difference matters
8 min liveFour sessions in, the Thornbury ward has become the standing example of a number that stopped being true. It would be easy, and wrong, to file it under tonight's heading. The ward was genuinely available before the outcome: a real fact about a real patient, recorded at the moment the prediction had to be made. It agreed with the outcome 90 percent of the time in the collection year and 50 percent after the reorganisation, and the model scored 0.949 held out and 0.765 shifted. That is a shortcut expiring, not information about the answer sneaking into the inputs. Two failures, two fixes, and a review that confuses them applies the wrong one.
LiveWhy the wrong fix is worse than no fix4 min▶
Run a stress test on a shortcut and you learn something real: the model leans on an arrangement, here is how much, here is what it costs when the arrangement moves. Run the same stress test on a leaked model and you learn nothing, because the leaked field carries the answer in every environment you can construct. It will pass the stress test beautifully. The team will report that the model is robust, and the word robust will now be doing active harm.
The tell is the timing, and only the timing. A shortcut is a real fact available before the outcome whose usefulness depends on an arrangement that can change. A leak is information that did not exist yet when the prediction had to be made. Everything else about them looks the same on a dashboard: a strong feature, a high score, a confident team. How to hunt one down inside a single model is a practitioner job and it belongs to learn-model-risk-with-phoebe p3.
Self-studyThe one question, and why it is the cheapest filter you have3 min▶
When a vendor, a paper or an internal team cites a published result, ask two things in one breath: what was the split, and could anything downstream of the outcome have been in the inputs. That is it. No statistics, no code, no methodology review. It takes about fifteen seconds to ask and it is the question that 294 papers did not get.
It is also usefully diagnostic in how it fails. A team with a clean answer gives it immediately, because they had to think about it while building. A team that has to go away and check has told you the question was never asked, which is a finding on its own and a perfectly civil place to leave it. Nobody has been accused of anything and everybody now knows where the evidence stands.
Put the Monday question to real evidence in the room
20 min build-alongNo laptop. Bring the external result you looked up for homework, plus whichever internal model your team is proudest of. Work both through the same six steps, in the same tone. The point of doing them together is that the checklist has to be something you can apply to your own work without flinching, or it will quietly become a weapon pointed outward only.
Name the prediction moment. In one sentence: at what point does this decision have to be made, and what is known at that point. Everything else in the exercise depends on this sentence being specific about time.
Ask what the split was. What was the model scored on, how was it separated from what it learned from, and who decided. "A random split" is an answer. "I would have to check" is a more informative one.
Ask whether anything downstream of the outcome could be in the inputs. Not accusingly, and not as a statistics question. Go feature by feature if the list is short, and ask when each one gets recorded.
Sort each suspicious feature into shortcut or leak. Available before the outcome and dependent on an arrangement is a shortcut. Existing only after the outcome is a leak. If the room cannot agree which, that feature is the finding.
Apply the matching fix, out loud. A stress test and a named watcher for the shortcut. A timing check on every field for the leak, handed to whoever owns the pipeline. Say which one you are choosing and why.
Write the three answers into the business case. The split, the timing, and what could not be established. The unanswerable ones are not a failure of the exercise; they are the honest boundary of what the evidence supports.
The one-sentence version to take into a meeting. "Before we lean on that result, what was the split, and could anything that only exists after the outcome have been in the inputs?" It is short enough to ask without a preamble, civil enough to ask a supplier, and it is the single highest-yield question in this whole track.
Three questions
Kapoor and Narayanan found leakage in 17 fields, collectively affecting 294 papers, across a taxonomy of eight types. What follows for a leader?
Nearly three hundred teams across seventeen disciplines did not independently become careless. The incentives, the review process and the tooling all permitted it, which makes the useful response procedural rather than a judgement about the authors. It also applies to your own team's results, which sit in the same population.
Thornbury's model leaned on the ward, which agreed with the outcome 90 percent of the time in the collection year and 50 percent after the reorganisation. Was that leakage?
The tell is timing. The ward was a real fact recorded at the moment the prediction had to be made, so the held-out number was an estimate of something. It just depended on an arrangement that changed. A leaked field would have passed the stress test instead of failing it, because it carries the answer in every environment you can build.
An internal team cites a published result to support a large investment. What single question filters the most risk for the least effort?
A leaked result generally replicates perfectly, because the same pipeline on the same data reproduces the flaw along with the number, so replication of that kind is reassurance about the wrong property. Peer review is weak here too, since the production of the evidence is described rather than shown. The split and timing question is the one that goes straight at the mechanism.
Before session 6
30 minTake the model or the result you are currently most confident about, and write one sentence describing what you would have to observe to stop believing it. Be concrete: a number on a named set, a behaviour in a named month, a comparison against a named alternative. If you cannot finish the sentence, write down why not. Session 6 is entirely about that sentence, and the ones people struggle to write are the interesting cases.