Where session 2 left the argument
Session 1 established that a held-out score travels on three assumptions, and session 2 broke the first one open: a pipeline returns a family of models, not a model, and the family agrees where you looked and disagrees where you did not. Tonight is the second assumption, the one about the set being an untouched draw. It is the only one of the three that is entirely under your own roof, which makes it the one worth measuring honestly rather than worrying about. Session 4 takes the same question outside the building and asks what a published benchmark licenses you to expect. The example is still Thornbury, the hospital group predicting which patients deteriorate within 24 hours.
★ What you walk out with tonight. The mechanism in one sentence, so you can spot it without a statistician: selecting on a set converts it from evidence into a target, and the reported number becomes the maximum of many noisy estimates. The measured cost at Thornbury: from one candidate to a hundred, selecting harder bought 0.0042 of reported accuracy and -0.0003 of real accuracy. And the lever that actually works, which is not discipline but size: the optimism falls from 0.0091 at a test set of 200 to 0.0030 at 2,000.
LiveWhat selection does to a measurement5 min▶
Every score computed on a finite set is an estimate with noise in it. Score one model and the noise is as likely to flatter it as to penalise it. Score twenty models and keep the best, and you have not just found the best model, you have also found the model the noise happened to favour most. Those two things arrive in the same number and cannot be separated afterwards.
That is the whole mechanism, and it needs no statistics to state: the reported number is the maximum of k noisy estimates, and the honest number is what that same model does on data nobody selected on. The gap between them is the optimism, and it is not a mistake anybody made. It is a property of having chosen.
The phrase people reach for here is Goodhart, and it fits, but tonight uses it in a narrower sense than usual. This is not about people gaming a target or a metric being badly designed; nobody at Thornbury is cheating. It is about what happens to the evidential standing of a measurement once the measurement starts selecting. Metric design and gaming belong to learn-okr-with-phoebe b9 and learn-model-evaluation-with-phoebe, which own the scorecard and the incentives.
The Thornbury set-up is deliberately plain. A pool of k candidate models, all scored on one test set of 400 patients. Keep the best, report its test accuracy. Then score that same model on 6,000 fresh patients nobody selected on, and compare. Each row of the table is the mean of 12 replications, so the numbers are not one lucky run.
Self-studyThe words used tonight4 min▶
- Test set. The set the final number is computed on. Tonight the interest is not in what it measures but in how many times it has been asked.
- Selection. Using a set to choose between candidates: models, thresholds, feature sets, stopping points. Any choice made by looking at the number counts.
- Reported accuracy. What the winning candidate scored on the set that picked it.
- Honest accuracy. What that same winning candidate scores on fresh data nobody selected on. The quantity everybody thinks the reported number is.
- Optimism. Reported minus honest. The measurable price of having chosen on the set.
- Standard error of the mean. How much a measured average would wobble if the whole exercise were repeated. When it is large relative to the differences you are comparing, the comparison cannot be made.
- Reusable holdout. A proposed remedy from Dwork et al. that uses ideas from differential privacy to let one set answer many questions with a stated budget. Named here, not derived.
Two lines, and only one of them moves
10 min liveThe sweep runs the same exercise at seven pool sizes, from one candidate to a hundred, and records both numbers each time. Reading it takes a few seconds, because the interesting thing is not the line that climbs. It is the line that does not.
LiveThe honest column does not move4 min▶
Read the teal line first. At one candidate the honest accuracy is 0.9500. At a hundred candidates it is 0.9497. Between those two points a team searched a hundred times harder and ended up with a model that is, in the world, very slightly worse. Everything gained by searching harder showed up in the reported number and nowhere else.
That is a stronger statement than "the number is inflated". It says the search was not buying the thing the search was supposed to buy. In this particular world all the candidates are close in real quality, so choosing between them on 400 patients is mostly choosing between noise realisations, and noise does not transfer.
The honest reading of the whole table: the reported number moved 0.0042 and the real model got 0.0003 worse. Whether that pattern holds in your setting depends on whether your candidates genuinely differ, which is a question you can ask and usually have not.
LiveWhy the gap stops growing4 min▶
The optimism runs 0.0027, 0.0045, 0.0049, 0.0053, 0.0068, 0.0072, 0.0072 as k goes 1, 2, 5, 10, 25, 50, 100. The first few candidates cost the most, and after that it flattens. Doubling from 50 to 100 bought nothing at all.
This matters for how you talk about it, because the intuition "more search means more inflation, without limit" is wrong and leads to the wrong controls. There is a ceiling, set by how far the luckiest plausible estimate can sit from the truth on a set of that size. Once your pool is big enough to find that luckiest realisation, adding candidates finds it again.
Note also the first row. At k=1, with no selection at all, the optimism is 0.0027 rather than zero, because the pool still had to be scored and the best of one is still the one you looked at. The zero point of this effect is not where people expect it to be.
Self-studyGoodhart, narrowly3 min▶
The familiar version of Goodhart is about incentives: name a target and people will optimise it at the expense of the thing it stood for. That version is about behaviour and belongs with metric design, which learn-okr-with-phoebe and learn-model-evaluation-with-phoebe own between them.
The version tonight is about evidence and needs no bad behaviour at all. A measurement that is used to choose stops being an independent observation of the thing chosen, because the act of choosing correlates the measurement with its own error. A model selection process run by scrupulous people on an honest set produces an inflated number, and the inflation is a property of the procedure rather than of anybody's conduct. That is why the useful response is measurement and set size rather than exhortation.
Size is the lever, and the field says stay calm
8 min liveIf the number of candidates is not the thing to control, something else is. Holding the pool at 60 candidates and varying only the size of the test set produces a clean answer, and one row of it carries a second lesson about what a small set can and cannot establish. Then the harder part of the evening: two strong empirical results that push back on the alarm the mechanism invites.
LiveBuy a bigger set, not more discipline4 min▶
Hold the pool at 60 candidates and change only the size of the test set. The optimism runs 0.0091 at 200 patients, 0.0062 at 400, 0.0044 at 1,000 and 0.0030 at 2,000. From 200 upward the fall is clean and well outside the error bars. The effect that could not be controlled by searching less can be controlled by measuring on more.
That is a procurement answer rather than a behavioural one, which is unusual and worth noticing. Asking a team to stop comparing candidates is asking them to do worse work. Asking for a larger set the final number is computed on costs data and a little patience, and it shrinks the problem directly.
The n = 100 bar is the exception, and it says something different. Its estimate is 0.0071 with a standard error of 0.0030, and it cannot be ranked against the 0.0091 at n = 200. At very small test sets even the measurement of the bias is too noisy to trust, so a small set does not merely give you a shaky accuracy, it also removes your ability to tell how shaky it is.
LiveThe disagreement, which is worth holding rather than settling5 min▶
The theory says repeated selection against a set inflates the number that set reports. Two strong empirical results say the effect is mild where it has been looked for hardest, and both deserve to be on the table.
Recht, Roelofs, Schmidt and Shankar (2019), "Do ImageNet Classifiers Generalize to ImageNet?" built new test sets from scratch and found accuracy drops of 3 to 15 percent on CIFAR-10 and 11 to 14 percent on ImageNet. Large drops, on the two most reused test sets in the field. Their conclusion about the cause is the part that matters here: the drops came from the models' limited ability to generalise to slightly harder images, not from test-set contamination through repeated use.
Roelofs, Shankar et al. (2019), "A Meta-Analysis of Overfitting in Machine Learning", NeurIPS 2019 examined over one hundred Kaggle competitions, where entrants score repeatedly against a public holdout and a separate set decides the final ranking. That is the mechanism running at industrial scale with real incentives, and they found little evidence of substantial adaptive overfitting.
So: a mechanism that is real and measurable, a measured cost at Thornbury that is under one accuracy point and saturates, and two field results finding the effect mild where it should be most visible. Those three are consistent with each other. Hold all three rather than choosing the one that makes the better slide.
LiveThe claim not to make4 min▶
It is tempting, having seen the mechanism, to conclude that leaderboards are broadly corrupted by reuse or that everyone is overfitting the benchmark. Neither is supportable. The measured optimism is under a point and stops growing, Recht et al. attributed the ImageNet and CIFAR-10 drops to a distribution gap rather than to reuse, and Roelofs et al. found little adaptive overfitting across a hundred competitions.
The defensible claim is narrower and more useful in a meeting: selecting on a set converts it from evidence into a target, the cost is measurable and modest, and it scales with how small the set is. That sentence survives every result on the table, tells you what to do, and does not require anybody to be exaggerating.
Where the fix research sits. Dwork et al. proposed the reusable holdout, which borrows machinery from differential privacy to let one set answer many questions within a stated budget. Worth knowing it exists and that the problem has serious attention. Not worth adopting on the strength of an effect this size, which is a judgement the measurement lets you make rather than guess.
Work out what your evaluation set has been asked to do
20 min build-alongNo laptop. You brought a count from session 2's homework: how many times one evaluation set has been looked at. Turn that count into an estimate and a decision. The aim is a right-sized response, which for most teams is a larger set and a retirement date rather than an apology.
Say the count out loud, and add the parts people forget. Models compared is the obvious one. Also count thresholds tuned on it, feature sets chosen with it, and every time a disappointing result sent somebody back to try again. That last category is usually the largest and the least recorded.
Say how big the set is. This is the number that determines the size of the effect, and in most teams it was chosen once, years ago, as a percentage of whatever data existed at the time.
Check whether the set can measure its own noise. If it is in the low hundreds of rows, you are in the n = 100 situation: the estimate is soft and so is any statement about how soft it is. Two candidates a point apart cannot be ranked on a set that size, and nobody has been told that.
Decide whether the pool genuinely differs. Ask whether the candidates being compared are meaningfully different approaches or twenty variations of one. When they are near-identical, selection is mostly picking a noise realisation and the honest number will not move, which is the Thornbury pattern.
Size the ask. Not "stop comparing models", which makes the work worse. Ask what it would cost to hold back four or five times as many rows for the final number. Data, labelling time, delay. Put a figure on it while the room is calm.
Write the retirement rule. One sentence: this set is retired and replaced after it has chosen between candidates N times, or after this date, whichever comes first. A set with an expiry is a set that stays evidence.
The one-sentence version to take into a meeting. "That number was chosen on the same data it was measured on, so it is optimistic by roughly the amount our set size implies; the fix is a bigger final set, not fewer experiments." A team that can say that has priced the effect. A team that either denies it or panics about it has not measured it.
Three questions
At Thornbury the reported accuracy rises from 0.9527 at one candidate to 0.9569 at a hundred, while the honest accuracy goes from 0.9500 to 0.9497. What is the strongest reading?
The honest column is flat across the whole sweep. In a world where the candidates are close in real quality, choosing between them on 400 patients is largely choosing between noise realisations, and noise does not travel to fresh data. Nothing was contaminated; the set was used honestly and still reported an optimistic figure.
The size sweep reports optimism of 0.0071 with a standard error of 0.0030 at a test set of 100, and 0.0091 with a standard error of 0.0022 at 200. What should be said about those two rows?
The interval on the 100 row is wide enough to overlap the 200 estimate comfortably, so no ordering between them is supported. That is the second lesson of the row and not a defect: a small set gives a soft number and also removes your ability to say how soft it is. From 200 upward the fall is clean and outside the error bars.
A colleague reads the k sweep and proposes telling the organisation that industry leaderboards are broadly corrupted by test-set reuse. What is wrong with that?
The mechanism is real and the field evidence says its practical size is small, and both belong on the table. The defensible claim is the narrow one: selecting on a set converts it from evidence into a target, the cost is measurable and modest, and it scales with how small the set is. Option C is also wrong, since Kaggle entrants score repeatedly against a public holdout, which is precisely where Roelofs et al. looked.
Before session 4
30 minFind one benchmark, leaderboard or published comparison that has been used in your area to justify a choice: a model, a vendor, a library, a direction. Write down two things about it. First, what task it actually scores, in the concrete: what the inputs are, who produced the labels, and when. Second, the sentence somebody used it to support. Bring both. Session 4 is about the distance between those two sentences, which is usually larger than anybody has checked.