The finding that did not survive

The bench trains twenty models on one data draw. The only difference between them is the random seed, which sets the initial weights and the order examples arrive in. Each model is scored twice: on 2,000 held-out patients from the collection year, and on 2,000 patients from the deployment year after the wards were reorganised. Twenty pairs of numbers, one correlation.

On the first draw that correlation was -0.89. Read plainly, it says the models that look best where you measured are the models that do worst where you did not, which would make held-out ranking actively misleading and would be the single most useful thing the bench could produce. The number was real, the code was right, and the finding was wrong.

• A seed. Initial weights and example order. Twenty seeds inside one draw give twenty models on identical data.
• A data draw. A fresh sample of the Thornbury world: 4,000 training patients, 2,000 held out, 2,000 shifted. The relationships in the generator do not change, only the sample does.
• What the correlation is computed over. The twenty models inside one draw. It is a statistic of twenty points, and it is reported as though it were a property of the pipeline.
• What stayed fixed across all five draws. The architecture of 3 inputs, 12 tanh hidden units and 1 logit, the training schedule, and the generator itself.
• The thing that was never varied in the first run. The data draw. The finding was a statement about the pipeline, computed on one sample, and nobody noticed the gap until a second sample existed.

The correlation is computed over twenty points, and the spread it is computed across is tiny: on draw 4 the twenty models sit inside a held-out spread of 0.003. Correlating a set of near-identical numbers against anything is a fragile operation, because the ordering of those numbers is close to arbitrary. The bench is not measuring a relationship between two quantities so much as reporting how twenty almost-tied models happened to line up in one sample.

That is why the ratio behaves differently. The ratio compares two spreads, which are aggregate quantities, and it is asking a question with a large effect behind it: models that agree to within thousandths where they were measured disagree by hundredths where they were not. Large effects on aggregate quantities survive resampling. Orderings among near-ties do not.

Reading one: there is nothing there. Held-out accuracy carries essentially no information about shifted accuracy, in either direction. The five values are scatter around no relationship, and the two strongly negative draws are what scatter looks like when you only run it twice.

Reading two: the estimate is underpowered. Twenty models per draw is a small sample for a correlation, so a weak true relationship could be present and invisible at this size. On this reading the bench has not shown there is nothing there; it has shown it cannot see.

The bench as it stands cannot separate these, and pretending otherwise would be the same error at a different level. What matters is that both readings forbid the original claim: neither one licenses "the best model on held-out data is the worst under shift". The way to separate them is more models per draw and more draws, which is a decision about what the answer is worth rather than a technical difficulty.

The instinct is to delete the dead claim and publish the survivor as though it had always been the point. That loses the most transferable thing in the episode. The write-up that helps a reader is the one that keeps the robust claim and reports the dead one next to it: here is what held in five draws out of five, here is what looked stronger on one draw and did not survive, and here is the number that killed it.

Reporting the death costs one sentence and buys two things. A reader who half-remembers the dramatic version from somewhere else is told directly that it did not replicate, and the next person to run this bench and see -0.89 on their first draw knows what they are looking at. Both of those are worth more than the appearance of having been right the first time.

Nothing in a normal review would have found this. The code was correct, the statistic was the right one, the data generator was documented, and a reader checking the arithmetic would have confirmed -0.89. Code review, a second pair of eyes and a reproducibility check all pass on a result computed from a single sample, because all three verify that the number is what the run produced.

The only thing that catches it is resampling, and resampling has to be a standing habit rather than a response to suspicion, because suspicion attaches to results that look wrong. This one looked right.

Five is enough to see a sign change and not enough to estimate anything precisely. It earns its place as a floor because it is cheap for a bench that runs in minutes, and because a claim that cannot survive five samples of its own data was never going to survive somebody else's. For a finding that would change a deployment decision, five is the point at which you start looking for a way to break it, not the point at which you stop.

The number also has to be reported. A range from five draws and the same range from fifty mean different things, and a reader cannot tell them apart unless the count is written next to the finding.