Where tonight sits
Sessions 1 to 4 asked what a held-out number can tell you about a world that moves. Tonight the world stays still and the number goes wrong anyway, because of something the analyst did rather than something the hospital did. The moment a set decides which model ships, it stops being a measurement of that model and becomes the thing the model was chosen to score well on. Session 6 measures the lever that controls the size of the effect.
★ What you walk out with tonight. The arithmetic behind selection optimism, which is the maximum of k noisy measurements rather than any kind of overfitting to training data. The measured size of it at Thornbury, which is smaller than the folklore and larger than zero. Three protections that cost nothing and are almost never in place. And a number to quote next to any reported score: how many candidates were compared on the set that produced it.
LiveThe setup, exactly as Bench B runs it4 min▶
A pool of k candidate models is trained on the collection year. Every one of them is scored on one test set of 400 patients. The analyst keeps whichever scored highest and reports that score. The honest number is that same kept model measured again on 6,000 fresh patients nobody selected on. Candidates are smaller networks than the rest of the course uses, 6 hidden units and 8 epochs, and every row of the sweep is the mean of 12 replications.
Nothing in that description is unusual practice. It is a hyperparameter sweep, a model bake-off, a leaderboard, or a week of iterating against the same evaluation, described plainly. The only thing Bench B adds is a second set that never took part in the choice, which is how the size of the difference becomes visible at all.
Self-studyThe words that will keep coming back4 min▶
- Reported accuracy. The winning candidate's score on the set that chose it. The number that reaches the deck.
- Honest accuracy. The same model, measured on data that took no part in the selection. What the model is actually worth.
- Optimism. Reported minus honest. At Thornbury it runs from 0.0027 at one candidate to 0.0072 at fifty and stops climbing.
- Selection, not overfitting. Nothing here is a model fitting noise in its training data. The candidates never see the test set during training; the analyst does.
- Saturation. The point past which more candidates stop buying reported accuracy. Between k = 50 and k = 100 the optimism does not move.
- Pre-registration. Writing down what will be selected on, and by which number, before any candidate exists.
The maximum of k noisy measurements
12 min liveAn accuracy computed on 400 patients is not the model's accuracy. It is the model's accuracy plus whatever those particular 400 patients happened to contribute. Score twenty near-identical candidates on the same 400 and you get twenty draws of that contribution. Keep the largest and you have kept the largest contribution too.
LiveWhy this is not overfitting in the usual sense4 min▶
Overfitting, as the word is normally used, is a model absorbing structure in its training data that does not exist in the world, and the cure is regularisation, more data or a smaller model. None of those touch what happens here. The candidates never see the test set while training. Every one of them generalises perfectly respectably. The inflation is introduced afterwards, by the person choosing, and it would happen just as reliably if the candidates were coin flips with no learning in them at all.
That is why regularising harder, cross-validating the candidates, or training them more carefully does not remove it. The optimism belongs to the selection step, so the only things that change it are how many comparisons the set was asked to arbitrate and how precisely it can arbitrate them.
LiveTwo things the gap grows with4 min▶
How many candidates. With one candidate there is no maximum to take, so the optimism is only the ordinary difference between two samples: 0.0027 at Thornbury. With a hundred, the analyst is taking the best of a hundred draws and the optimism reaches 0.0072.
How noisy each measurement is. The maximum of a hundred tightly clustered estimates is barely above their middle. The maximum of a hundred widely scattered estimates is far above it. Anything that widens the scatter, and the size of the test set is the main thing that does, widens the gap. Session 6 measures that directly.
A team runs a 60-configuration grid search, keeps the configuration with the best score on the evaluation set, and reports that score in the model card. Nobody logged the other 59 numbers and nobody recorded that the set was asked to rank 60 things. The reported score is not wrong, and it is not the score of the model that shipped either.
The honest column does not move
10 min liveTwo columns, seven rows, and the whole argument sits in the second column. One of them climbs with every extra candidate. The other one was measured on patients nobody selected on, and it is flat.
| Candidates k | Reported | Honest | Optimism |
|---|---|---|---|
| 1 | 0.9527 | 0.9500 | 0.0027 |
| 2 | 0.9544 | 0.9499 | 0.0045 |
| 5 | 0.9548 | 0.9499 | 0.0049 |
| 10 | 0.9552 | 0.9499 | 0.0053 |
| 25 | 0.9562 | 0.9494 | 0.0068 |
| 50 | 0.9567 | 0.9495 | 0.0072 |
| 100 | 0.9569 | 0.9497 | 0.0072 |
LiveWhy the flat column is the finding4 min▶
A climbing reported column on its own proves nothing. It is entirely possible that searching a hundred candidates finds a genuinely better model than searching one, and if it did, both columns would climb together. They do not. The honest column reads 0.9500 at one candidate and 0.9497 at a hundred, which is to say that a hundred times more search produced a model that is very slightly worse on data nobody chose it with.
That is the observation worth taking away, because it is the one that separates a real improvement from a selected one. Any time a search is reported as having improved a model, the question is whether the improvement shows up on something the search never touched. If there is no such measurement, there is no way to tell the two apart, and the default assumption should not be that the search got lucky twice.
Self-studyThe saturation, and why it is reassuring3 min▶
The optimism climbs steeply from k = 1 to k = 10, more slowly to k = 50, and not at all from k = 50 to k = 100. Both of the last two rows read 0.0072. The maximum of many draws grows very slowly once there are already many draws, so doubling a search that is already large barely moves the reported number.
The practical consequence is worth stating plainly: the difference between comparing 50 candidates and comparing 5,000 is far smaller than the difference between comparing 1 and comparing 10. Most of the damage is done early. A team that says "we only tried a handful" has not said what they think they have said.
Three things that cost nothing
8 min liveNone of these require more data, a different model or a statistician. All three are process, and all three are usually missing from the places the number ends up.
Split three ways before anything is trained. Training, validation, and a set that is physically separate and has a name that says so. The point of the third one is that it has never answered a question.
Write the selection rule into the plan. Which metric, on which set, with ties broken how. A rule written after the candidates exist is a description of the winner, not a rule.
Keep a look counter next to the set. One line in a log per decision the set arbitrated: a threshold, a feature set, an architecture, an early-stop point. The count is what the reader needs in order to discount the number.
Report k with the score. "0.956 on 400 held-out patients, best of 60 configurations" is a different claim from "0.956", and it takes six extra words.
Measure the honest number when you can afford it. If a genuinely untouched set exists, score the chosen model on it once and report both figures. The difference between them is your own optimism, measured rather than assumed.
Treat a reused set as spent. Once a set has chosen a model, it can still describe that model, but it can no longer rank a new one against it without the same inflation returning.
The habit to keep. Whenever a score arrives without a candidate count, ask for one. Not to catch anybody out, but because the two numbers mean different things and only one of them is on the slide. A reviewer who knows that 60 configurations were compared on 400 rows knows roughly how much of the score to spend, and a reviewer who does not is reading a maximum as if it were a measurement.
Three true things that pull against each other
6 min liveThe mechanism is arithmetic and is not in doubt. The measurement here is small. And two strong field results found the effect mild where it was expected to be worst. All three hold, and the honest position is to keep all three rather than pick the one that suits the argument.
LiveWhat the two field results actually found4 min▶
Recht, Roelofs, Schmidt and Shankar built new test sets for CIFAR-10 and ImageNet and measured accuracy drops of 3 to 15 percent on CIFAR-10 and 11 to 14 percent on ImageNet. The drops were large. Their conclusion about the cause is the part that matters here: they attributed them to the models' limited ability to generalise to slightly harder images rather than to contamination of the test set through repeated use. They also observed that accuracy gains on the original test sets translate to larger gains on the new ones.
Roelofs, Shankar and colleagues went at it from the other direction, across over one hundred Kaggle competitions where entrants repeatedly score against a public holdout and a separate set decides the final ranking. That is adaptive analysis at industrial scale, with money on it. They found little evidence of substantial adaptive overfitting.
Self-studyThe claim that is defensible3 min▶
Two sentences are unsupportable in opposite directions. "Everyone is overfitting the benchmark" is not supported by anything: the measured effect here saturates under one accuracy point, and the two strongest field studies looked for large adaptive overfitting and did not find it. "Test-set reuse is harmless" is not supported either: the mechanism is arithmetic, the gap is measurable, and it grows with both the number of comparisons and the noisiness of each one.
What survives all three positions is narrower and more useful. Selecting on a set converts it from evidence into a target. The cost of that conversion is real, measurable and modest, and it scales with how small the set is. That is a claim you can defend in a review without either alarming people or reassuring them falsely.
Three questions
Reported accuracy rises from 0.9527 at k = 1 to 0.9569 at k = 100 while honest accuracy reads 0.9500 and 0.9497. What does the flat honest column establish?
If the larger search had found a better model, the honest column would have moved with the reported one. It did not: 0.9500 down to 0.9497 across a hundredfold increase in search. The maximum of many noisy estimates grows; the thing being estimated does not.
A team regularises every candidate harder and re-runs the same 60-configuration sweep against the same evaluation set. What happens to the selection optimism?
The candidates never see the test set while training, so nothing about how they were fitted changes the arithmetic of taking a maximum over 60 noisy measurements. The two levers are the number of comparisons and the precision of each one, which is the test-set size.
Which sentence about test-set reuse is defensible given the mechanism, Bench B and the field evidence together?
The first overstates what anybody has measured: the optimism here saturates under one accuracy point, Recht et al. attributed the ImageNet and CIFAR-10 drops to harder images rather than reuse, and Roelofs et al. found little adaptive overfitting across over a hundred competitions. The third denies an arithmetic mechanism that is visible in the k sweep. Only the middle sentence holds all three positions at once.
Before session 6
30 minTake a score your team has reported in the last quarter and reconstruct two numbers about it: how many candidates were compared on the set that produced it, and how many rows that set had. Ask whoever ran it; the answer is usually available and almost never written down. Then predict, in writing and before session 6, what would happen to the optimism if the same sweep had used a quarter as many test rows. Session 6 measures exactly that, and a prediction made beforehand is the only kind worth anything.