Where tonight sits
Session 5 showed that selecting on a set inflates the number it reports, and named two things the gap grows with: how many candidates were compared, and how noisy each measurement was. Tonight is the second one, because it is the one under your control at the moment you split the data. A test set has two jobs, estimating an accuracy and ranking things against each other, and a small one fails at both in ways that reinforce each other.
★ What you walk out with tonight. The size sweep with its error bars, so you know the shape of the relationship rather than a rule of thumb. A working method for turning a claimed difference into the number of rows it needs, which takes about four lines. The reason a 0.4 point margin is not a finding on a small set. And the observation at n = 100 that is worth more than the rest of the table: an estimate that cannot even rank itself.
LiveTwo jobs, and a small set fails at both4 min▶
The first job is estimation. A test set answers "roughly how accurate is this model", and the answer arrives with a width around it that comes from having counted a finite number of patients. Nobody disputes this and almost nobody reports it.
The second job is ranking. A test set answers "is this model better than that one", and that question is harder, because it needs the width around the difference rather than the width around either number. A set that gives a usable answer to the first question can be useless for the second.
Session 5 added a third thing a small set does, which is not a failure of the set so much as a consequence of it: selection takes the maximum of noisy measurements, and a noisier measurement gives selection more to work with. The three compound.
Self-studyThe words that will keep coming back4 min▶
- Standard error. The typical distance between an estimate and the thing it estimates, from having counted a finite sample rather than the whole world.
- Interval. The band around an estimate that is consistent with what was counted. Reporting the point and not the band is the normal practice and the normal mistake.
- The square-root law. The noise on an estimate falls with the square root of the row count, so halving it costs four times the rows.
- Ranking against estimating. Two different jobs with two different size requirements. Ranking always needs more.
- The compounding. A noisier estimate widens the interval and enlarges the selection optimism at the same time. Session 5 measured the second effect at a fixed size; tonight moves the size.
- Precision against validity. A bigger set narrows the interval around today's accuracy and says nothing about whether tomorrow resembles today. Session 1 owns that distinction and it does not go away tonight.
The same bias, measured at five sizes
12 min liveCandidate count fixed at 60, forty replications per row, and the only thing changing is how many patients the test set holds. The bars are the error bars on the measurement of the bias itself, which is the part of this table that does the most teaching.
| Test n | Optimism | SE |
|---|---|---|
| 100 | 0.0071 | 0.0030 |
| 200 | 0.0091 | 0.0022 |
| 400 | 0.0062 | 0.0017 |
| 1000 | 0.0044 | 0.0010 |
| 2000 | 0.0030 | 0.0006 |
LiveThe row that teaches the most is the one that misbehaves4 min▶
Read the table from 2000 upwards and it is tidy: 0.0030, 0.0044, 0.0062, 0.0091, with error bars small enough that the ordering is not in question. A bigger test set buys back the optimism, cleanly and monotonically. Then the last row breaks the pattern, and it breaks it downwards, reading 0.0071 where the trend wanted something larger than 0.0091.
The standard error is what to look at, not the point. At n = 100 it is 0.0030, the largest in the table, and wide enough that the estimate overlaps the one at n = 200. The honest statement is that these two rows cannot be ranked against each other. That is not a defect in the bench; it is the subject of the session arriving one level up. At 100 rows the measurement of the bias is as unreliable as the accuracies the bias was measured from, and a run that reported only the point estimate would have produced an apparently interesting reversal out of nothing.
Self-studyWhy halving the noise costs four times the rows3 min▶
An accuracy is a proportion counted over independent rows, and the noise on a counted proportion falls with the square root of the row count rather than with the row count itself. Going from 500 rows to 1000 does not halve the width of the interval; it shrinks it by about a third. Halving the width needs four times the rows, and quartering it needs sixteen times.
That is why the answer to "should we grow the test set" is so often no in practice and yes in principle. The next increment of precision is always more expensive than the last one, and the returns are visible in the SE column: 0.0030, 0.0022, 0.0017, 0.0010, 0.0006 while the row count goes up twentyfold.
Beating them by 0.4 points is not a finding
10 min liveThe most common claim in a model review is a small margin over an incumbent. It is also the claim most likely to be manufactured by the process that produced it, because a margin of that size is exactly what selection alone buys.
LiveWhy the difference is noisier than either number4 min▶
Two accuracies measured on independent test sets each carry their own noise, and the difference between them carries both. That is the geometry people skip: a margin needs more rows than a point estimate does, not fewer, even though a margin feels like a smaller quantity.
The one saving grace is that scoring both models on the same rows removes part of the problem, because patients that are hard for one model are often hard for the other and that shared difficulty cancels out of the difference. It is a real and worthwhile gain, and it is also the reason the same set then ends up arbitrating every comparison anybody makes, which is session 5's problem returning by the back door.
LiveWorking backwards to the number of rows4 min▶
Start from the decision rather than from the data. Name the smallest difference that would change what anybody does. If a two-point improvement is the threshold for replacing an incumbent, then a set sized to resolve two points is enough and a set sized to resolve a tenth of a point is a waste of rows that would have trained a better model. If the decision genuinely turns on a tenth of a point, the square-root law says what that costs and the honest answer is often that the data does not exist.
Doing this in advance has a second effect that matters more than the arithmetic. A team that wrote down "we need to see at least two points" before the run cannot later be impressed by half a point, which is the failure mode this whole track is built around.
A small test set is dangerous twice
6 min liveThe two failures are not independent problems that happen to co-occur. They have the same cause, which is why the row count is a better lever than anything downstream of it.
Run the size sweep, then size your own set
14 min build-alongTwo blocks against the engine on this page. The first reproduces the size sweep including its error bars, which is the only way the n = 100 row means anything. The second turns a claimed difference into a row count.
Open the console on this page. ep-live.js is loaded here, so window.THORNBURY already exists with the generator and the selection helpers on it.
Run the sweep with the error bars in it. A mean over replications without its standard error is what produced the apparent reversal at n = 100. Compute both or compute neither.
Check the two rows that overlap. Print the interval at n = 100 and at n = 200 and satisfy yourself that they cannot be ranked.
Measure the noise on a single accuracy. One model, five test-set sizes, the standard error at each. Watch it fall with the square root rather than with the row count.
Size a set from a claimed difference. Feed the smallest margin your team would act on into the last function and see the row count it asks for.
Compare it with the set you actually have. If the required count is larger than the rows in hand, the margin was never measurable and the honest report says so instead of quoting it.
The habit to keep. Two numbers belong next to every reported accuracy, and they cost one line: the number of rows it was counted on, and the number of candidates that were compared on those rows. With both, a reader can work out roughly how much of the score to believe. With neither, the score is a point estimate of unknown precision selected by an unknown amount of search, and no amount of decimal places fixes that.
Three questions
The measured optimism reads 0.0091 at n = 200 and 0.0071 at n = 100, which breaks the trend. What is the right reading?
The standard error at n = 100 is the largest in the table and wide enough that the two rows overlap. Reading the point estimates alone would turn noise into a reversal, which is the same error the rest of the sweep is warning about, arriving one level up.
A team reports that their model beats the incumbent by 0.4 accuracy points, after keeping the best of 60 configurations on a test set of 200 rows. What can be concluded?
Selecting the best of 100 candidates moved reported accuracy by 0.0042 while real accuracy moved -0.0003, so a margin of that magnitude is exactly what a search of that size manufactures. At 200 rows the optimism was measured at 0.0091, which is larger than the claimed margin.
Doubling a test set from 500 rows to 1000 does what to the width of the interval around the accuracy?
An accuracy is a proportion counted over rows and its noise falls with the square root of the count. Halving the width needs four times the rows and quartering it needs sixteen times, which is why the next increment of precision always costs more than the last one.
Before session 7
30 minTake the score you reconstructed for session 5 and finish the job: write down the row count of the set it was measured on, the candidate count that was compared on it, and the smallest difference your team would act on. Then run block 2 and see whether the set you have can resolve the difference you care about. Bring one sentence saying either what margin your current set can defend, or how many rows you would need to defend the one you have been quoting. Session 7 moves to numbers somebody else produced, where you will not be able to ask any of these questions.