What the benches measure
Two quantities that are invisible from any single accuracy figure: how much models that agree on the test set disagree off it, and how much of a reported number was bought by selecting on the set that produced it. Both are computed by training networks when you click, on data whose true relationships are written into the generator, so what the models found can be compared with what was actually there.
★ What you walk out with tonight. The spread you can produce from one pipeline by changing nothing but a seed, run on the draw of your choosing. The measured price of selecting on a test set, and the test-set size that buys it back. And one habit worth more than either: writing the falsifier before the run, so the result has something to fail against.
LiveWhere a known answer comes from3 min▶
Thornbury's patients are generated from relationships written out in ep-live.js. Two signals genuinely cause deterioration: a vitals drift and a lab trend. One feature does not cause it at all: the ward. In the collection year the high-acuity ward received most of the patients who deteriorated, so the ward agrees with the outcome about 90 percent of the time. After the reorganisation it agrees about 50 percent of the time.
That construction is what makes the benches answerable. In a real deployment you never learn whether a model leaned on a mechanism or an arrangement, because you only see the score. Here the arrangement is a line of code, so the question has a checkable answer.
Thornbury's review approved on 0.949 held out. Nothing in that process was careless: the split was honest, the metric was sensible, the model was competent. The review simply had no way to ask which of the two kinds of pattern the model had found, and the deployment year answered the question for it at 0.765.
Self-studyWhat is real here, and what is not3 min▶
- The networks. Three inputs, twelve tanh hidden units, one logit, trained by stochastic gradient descent in the page. Not a stored curve.
- The only difference between the twenty. The random seed, which sets the initial weights and the order examples arrive in. Same data, same architecture, same schedule.
- Every accuracy. Counted on data the model did not train on.
- Reliance on the ward. Measured behaviourally: flip the ward on every patient, hold everything else fixed, count how many predictions change.
- What is not modelled. No human behaviour, no simulated results, and no finding written in advance. A number here that disagreed with the course notes would be the notes' problem.
Two questions, two panes
10 min liveThe panes answer different halves of the same worry. Pane A asks whether the number identifies the model. Pane B asks whether the number was earned or selected. A model can pass one and fail the other, and most review processes check neither.
LiveThe result that is robust, and the one that is not4 min▶
Run pane A across all five draws and one thing holds every time: the shifted spread is wider than the held-out spread, by between 2.7 and 11.3 times. Twenty models agree to within a few thousandths where they were measured and disagree by three to five hundredths where they were not.
One thing does not hold. The correlation between held-out and shifted accuracy reads -0.89 on the first draw, +0.03 on the second, -0.87 on the third, +0.09 on the fourth and -0.75 on the fifth. The sign moves with the draw, which means held-out accuracy neither predicts nor reliably anti-predicts how a model will do under shift. It is close to uninformative about it, and the temptation to read the first draw as "the best model on paper is the worst in practice" is exactly the mistake session 9 is about.
Self-studyWhy pane B's honest column is the one to watch3 min▶
The reported number climbs as the candidate count rises: 0.9527 at one candidate, 0.9569 at a hundred. The honest number, measured on six thousand patients nobody selected on, sits at 0.9500 and 0.9497. Selecting a hundred times harder bought about four thousandths of reported accuracy and nothing at all of the real thing.
That is the whole mechanism in two columns, and it is also why the effect is easy to overstate. The gap saturates and never reaches a full accuracy point at these sizes, which is consistent with what Recht et al. and Roelofs et al. found in the field. The honest summary is that selecting on a set converts it from evidence into a target, at a cost that is real, measurable and modest.
Run both panes
26 min build-alongBefore you click anything, write down what you expect. Then run pane A on draw 1, read the correlation, and run it again on draw 4. The second run is the point of the session.
Write the falsifier first, every time. Before pane A runs, finish this sentence on paper: "if the shifted spread comes out below ___, seed variation is not worth a control." Then run it. A prediction made after the number appears is not a prediction, and the habit of writing one down is the cheapest protection against reading whatever the data happens to show as confirmation.
Turn two benches into one paragraph
6 min liveWhat a review can act on is not a number, it is a sentence somebody else can check. Four parts, and the fourth is the one nobody writes.
State the score and its world. The figure, the data it was computed on, and when that data was collected. "0.949 on held-out patients from the collection year" rather than "94.9 percent accurate".
State what it depends on. The resemblance that has to keep holding. At Thornbury that is the ward arrangement, and naming it is the difference between a monitored model and a surprised one.
State what you did not measure. The stress you did not run, the population you have no data for, the seed you did not vary. Session 4 is about what a stress test can and cannot establish, and the answer always includes a list of things it could not.
State the falsifier and the watcher. What observation would make you withdraw the claim, who would see it, and how long that would take. A claim with no falsifier is not a weak claim; it is not a claim.
Three questions
Pane A returns a correlation of -0.89 between held-out and shifted accuracy on draw 1. What may you write down?
The sign is not stable across draws, so the only defensible reading is that held-out accuracy carries almost no information about shifted accuracy in either direction. A single draw producing a strong correlation is what a coincidence looks like, and it is the exact error session 9 documents.
In pane B the reported accuracy rises from 0.9527 to 0.9569 as candidates go from 1 to 100, while the honest accuracy stays near 0.9500. What happened?
Selecting the maximum of many noisy estimates inflates the estimate, not the thing being estimated. The flat honest column is the evidence: the model you end up shipping is no better, it just measured better on the set you chose it with.
Which sentence belongs in the write-up and is almost always missing?
The method and the interval are usually present and neither tells a reader when to stop believing the result. A falsifier with an owner and a detection time converts a score into a claim somebody can check, which is the only part of the write-up that survives the model being wrong.
After the track
45 minTake one model your team owns and run the cheapest version of pane A on it: retrain it three times changing only the seed, and compare the three on any slice you did not select on. Three runs is enough to see whether the spread is thousandths or hundredths. Write the four-part paragraph from Part 3 underneath, including the falsifier, and give it to somebody who was not involved. If they cannot tell you what would make the claim false, the paragraph is not finished.