Where tonight sits
Session 7 took one published number apart and wrote down what it licensed. Tonight the unit is a literature rather than a number, because the flaw in question was found 294 times across 17 disciplines and a flaw distributed that widely is a property of the conditions rather than of any team. Session 9 turns the same suspicion inward, on a finding produced right here that did not survive one more data draw.
★ What you walk out with tonight. Three numbers you can cite from memory: 17 fields, 294 papers, eight types. A leak you built yourself, running against the same engine the rest of the track uses, so you have seen a stress test certify a broken model rather than catch it. A split between the checks on somebody else's work that are free and the checks that no reviewer has ever been in a position to run. And a four-rung reading order that puts the free ones first.
LiveWhat tonight is about, and what it deliberately is not4 min▶
Leakage is information about the answer reaching the model's inputs when it should not have. The held-out score then measures something other than the ability to predict, and it measures it very well, which is the problem. Hunting one down inside a single validation is a separate job with its own timing analysis and its own filing, and it belongs to learn-model-risk-with-phoebe p3.
The subject here is evidence at the scale of a literature. Kapoor and Narayanan's survey reports leakage in 17 fields, collectively affecting 294 papers, across a taxonomy of eight types spanning textbook errors through to open research problems. That range is the part that gets dropped in retellings and it is the part a practitioner needs: some of the eight are mistakes anybody can be taught to avoid in an afternoon, and some are not solved by anyone.
Self-studyThe words that will keep coming back4 min▶
- Leakage. Information about the answer reaching the inputs. The reported score stops being an estimate of what the model can do on a case whose outcome is not yet known.
- The prediction moment. The instant the decision has to be made. Every input has to have existed before it, and dating each input against it is the whole check.
- Shortcut. A feature that predicts without causing, genuinely available before the outcome. Thornbury's ward. A stress test is the fix.
- A structural failure. One produced by shared conditions rather than individual error, so it is predicted by the conditions and not by the person.
- The printed record. What a reader actually has: a methods section, a feature list, sometimes code. Much smaller than what would be needed to verify the claim.
- Cost order. Running checks cheapest first, because the cheap ones catch most of it and the expensive one is the only one anybody ever schedules.
Two hundred and ninety-four is not a story about care
10 min liveThe first instinct on hearing the number is to form a view about the authors. It is the wrong model of what happened, and more usefully it leads to a fix nobody can implement. The alternative reading changes what you do on a Monday.
LiveWhy rerunning it proves nothing4 min▶
The instinct when a result looks too good is to try to reproduce it. That instinct is well trained and, for this particular flaw, it points at the wrong property. Running the same pipeline on the same data reproduces the flaw with perfect fidelity, along with the excellent number that came out of it. A leaked result generally replicates. Everybody who runs it gets the same figure, and the figure is still not an estimate of what the model can do on a case whose outcome is unknown.
This is why an independent replication is reassurance about the wrong thing, and it is also why the checks that matter are all about provenance rather than arithmetic. Nothing in the numbers is wrong. What is wrong is where one of the columns came from, and no amount of recomputation reaches that.
A supplier answers a query about an unusually high accuracy by offering to run the evaluation again with your team watching. That is a courteous offer and it settles nothing. The question that settles something is which columns went into the model and when each of them is written, and it can be answered over email in a paragraph.
Build the leak, then watch the stress test approve it
13 min liveThornbury already has an honest failure in it: the ward predicts without causing, and the shifted set exposes it. Add one more column, written after the outcome and joined back onto the row, and the same stress test that caught the ward reports that everything is fine.
LiveRead the second line of output before anything else4 min▶
The honest model does what the previous sessions trained you to expect: a strong held-out figure and a weaker shifted one, because the ward stopped agreeing with the outcome when the wards were reorganised. The gap between those two numbers is what a stress test is for, and it is doing its job.
The leaked model breaks the pattern in the direction nobody looks for. Both of its numbers are strong, because the extra column was built to agree with the outcome 95 percent of the time in every set it appears in, deployment year included. A reorganisation of the wards cannot touch a column that the outcome itself wrote. The model that is unfit for any use is the one that passes the robustness check cleanly, and it will pass every other environment you can construct too.
The two reliance numbers finish the picture. Session 2 measured the honest model flipping its mind on 7 patients in 10 or more when only the ward changed. Run the same flip on the leaked model and that reliance collapses, because it has found a better column to read; run the flip on the leaked column instead and the dependence has moved there. On a dashboard the first half of that reads as a model that stopped depending on a fragile feature, which is exactly what everybody has been asking it to do.
Self-studyWhy this arrives without anyone deciding to do it3 min▶
Nobody writes the line above on purpose. The realistic version is a join: a table of patients, a table of events, one key, and a merge that pulls in whatever the events table holds for that patient. Some of what it holds was written during the episode and some was written when the episode closed, and the merge has no opinion about which. The resulting frame looks entirely ordinary, with one column per thing you might want to know.
The reason a timing check has to be run per column rather than per table is sitting in that sentence. A table is not early or late. Individual columns are, and the ones that are late are usually the informative ones, which is why they survive feature selection so comfortably. Tracking a suspected leak back through a pipeline to the join that produced it is the practitioner job owned by learn-model-risk-with-phoebe p3.
What a reviewer could have caught, and what nobody could
8 min liveA reader has a methods section, a feature list and sometimes released code. That is a much smaller object than the thing that produced the number, and the boundary between the two is fixed. Knowing exactly where it sits stops both the wasted effort and the false comfort.
LiveThe one on the left that people skip4 min▶
Of the four checkable questions, the split unit is the one that gets waved through. A sentence saying the data were split into training and test does not say what was split: rows, patients, episodes, sites, weeks. Those are five different experiments and four of them can put the same underlying thing on both sides. The sentence looks complete because it uses the right words, and a reader who is scanning for the word split finds it and moves on.
The check is to read that sentence and ask which noun it names. If it names no noun, write that down rather than assuming the obvious one. Assuming the obvious one is how a reader converts a gap in the record into a fact, and doing it silently means nobody downstream can tell it happened.
Self-studyWhy the right panel is the structural half3 min▶
Peer review is strong at catching claims that do not follow from the evidence reported, and that strength is real. It has no purchase at all on how the evidence was produced, because that part arrives as a paragraph of prose written by the people who produced it. A reviewer reads that a held-out set was used and has nothing to check the sentence against. Nobody has behaved badly and no amount of additional care changes the shape of the problem.
That is what makes the finding structural rather than a story about three hundred teams. The conditions, the incentives and the format of the record all permitted it, and the same conditions currently apply to whatever your own team is about to publish or ship. A filter applied only to external evidence has a hole in the middle, and the hole is where the work you are most invested in sits.
Cheapest first, because the cheap ones catch most of it
8 min liveThe order matters more than the list. Run the free checks on everything, always, and reserve the expensive one for the claims a real decision rests on. Reversing that order is how teams end up doing a thorough job on one paper a year and nothing on the rest.
Write the prediction moment down first. One sentence: at what instant does the decision have to be made, and what is known then. Every later question is measured against this sentence, so vagueness here makes the rest of the pass meaningless.
Quote the split sentence verbatim. Copy it out of the paper rather than summarising it. Summarising is where a missing noun gets silently filled in with the one you expected.
Date the inputs, one column at a time. A table is neither early nor late; columns are. Mark each one as before, after, or unknown. The unknown ones are the whole output of this step.
Ask the entity question in writing. Can the same patient, episode or site appear on both sides of the split under different identifiers. To a supplier this is a one-paragraph email, and the speed of the reply tells you whether it had been considered.
Record what could not be established. Not as a failure of the review. It is the honest boundary of what the evidence supports, and it is the part that a decision made six months later actually needs.
Run the same four rungs on your own most-cited result. Same order, same wording, no softer. A checklist that is only ever pointed outward stops being a checklist and becomes a rhetorical device.
Schedule the expensive rung for exactly one claim. The one carrying the largest decision. Naming the owner and the date is what separates it from an intention.
The habit to keep. When a result is unusually good, the first question is not how it was validated but when each of its inputs was written. Validation questions get answered with confidence because the team thought hard about validation. Timing questions get answered with a pause, and the pause is the finding.
Three questions
The leaked Thornbury model scores strongly on the held-out set and strongly on the shifted set as well. What does that pattern tell you?
The extra column was constructed to agree with the outcome in both years, so no reorganisation of the wards touches it and no harder scenario would either. A stress test repairs a shortcut, whose usefulness depends on an arrangement that can change, and certifies a leak, which does not depend on any arrangement at all.
A supplier offers to rerun their evaluation with your team watching, to settle a question about an unusually high accuracy. How much does that settle?
A leaked result generally replicates perfectly. Every run gives the same excellent figure, and the figure is still not an estimate of what the model can do on a case whose outcome is unknown. The checks that reach this are about where a column came from and when it was written, not about recomputation.
Kapoor and Narayanan found leakage in 17 fields, affecting 294 papers, across a taxonomy of eight types. Why does the scale change the procedure rather than only the mood?
Nearly three hundred teams across seventeen disciplines did not independently become careless, so the explanation has to be in the incentives, the review format and the tooling they shared. Vigilance cannot be implemented; a short ordered list of checks can, and it survives tired people and staff changes. Those same conditions also apply to your own results.
Before session 9
30 minRun the four rungs on your own team's most-cited internal result, in order, and stop after the third. Write the prediction moment as one sentence, quote the split sentence verbatim from wherever it is recorded, mark every input as before, after or unknown, and send the entity question to whoever owns the pipeline. Then write one line predicting what the unknown column will turn out to be. Session 9 is about findings that did not survive a second look, and a prediction written before the answer arrives is the only way to know whether you were reading the evidence or your own expectations.