Twenty seeds, one pipeline

D'Amour et al. (2020) define it in one sentence: "An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain." Read it as a property of the pipeline rather than of any model that came out of it. The pipeline is the data, the architecture, the loss, the optimiser and the schedule. Run it twice with different seeds and it hands you two different predictors. If those two score the same held out, the pipeline did not determine which one you got, and the held-out score cannot tell you which one you are holding.

The operational test is short. Vary something that should not matter, hold everything that should matter fixed, and score the results in more than one place. At Thornbury the thing that should not matter is the random seed, which sets the initial weights and the order the examples arrive in. Nothing else moves: same 4,000 training patients, same three inputs into twelve tanh hidden units into one logit, same schedule.

The same authors are blunt about the consequence: "Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains."

• Underspecification. A pipeline that can return many predictors with equally strong held-out performance, which then behave differently in deployment. A property of the pipeline.
• Seed sweep. Running the same pipeline n times with only the random seed changed, and keeping every score rather than the best one.
• Spread. Highest minus lowest across the sweep. Blunt on purpose: it is the quantity a reviewer can check by eye against a printed list.
• The ratio. Shifted spread divided by held-out spread. How much wider the models are where nobody measured them than where everybody did.
• Data draw. A regenerated world: new training, held-out and shifted patients from the same recipe. Five of them exist, and a finding that appears in one is not yet a finding.
• The standing pair. Mean held-out 0.949 and mean shifted 0.765 across the five draws.

Fixing the seed is good hygiene and it is the standard advice, because it makes a result reproduce. What it does not do is make the result representative. A fixed seed picks one draw from the distribution the pipeline can produce and then hides the fact that there was a distribution, because every rerun returns the same point and the printout looks stable.

The two habits are not in conflict, they answer different questions. Fix the seed when you want a build to be byte-identical tomorrow. Sweep the seed when you want to know what your pipeline is capable of returning. A team that only ever does the first has a reproducible number and no idea how much of it was luck.

Session 2's measurement runs alongside the two accuracies for free, because it needs nothing but predictions. Across twenty seeds at Thornbury the reliance on the ward runs from 0.70 to 1.00: some models change their mind about 7 patients in 10 when only the ward moves, others about all of them.

That column is the mechanism underneath the spread. The models that lean hardest on the arrangement are the ones with the most to lose when it goes, and the held-out score is blind to the difference because the arrangement still holds on the held-out rows. Three columns, two of which move and one of which does not, is the whole finding in a printout you can put on one slide.

Two things only: the initial weights, and the order the training examples arrive in. Both are usually described as implementation details, and both are outside the specification of what the model is supposed to be. Nobody writes a model card that says which permutation of the training set was used.

That is exactly what makes the sweep convincing. If a quantity nobody considers part of the model produces a 0.051 spread in deployment accuracy, then the specification is not pinning down the thing that matters. A larger perturbation, a different architecture or a different feature set would produce a bigger spread and prove much less, because those are things the specification was never claiming to fix.

In five draws out of five the shifted spread was larger than the held-out spread, by 2.7, 4.8, 4.0, 11.3 and 3.0 times. The absolute numbers are worth holding too: the held-out spreads are thousandths, from 0.003 to 0.019, and the shifted spreads are hundredths, from 0.034 to 0.051. Twenty models that agree to within three thousandths of an accuracy point on draw 4 disagree by thirty-four thousandths once the ward arrangement changes.

Nothing here required an unusual world or an adversarial perturbation. The training set is 4,000 patients with two genuine signals in it, the model is small, and the shift is one arrangement in a hospital group changing. The spread is what the pipeline was always able to produce; the shifted set is only the place where it becomes visible.

The correlation between held-out and shifted accuracy reads -0.89 on draw 1, +0.03 on draw 2, -0.87 on draw 3, +0.09 on draw 4 and -0.75 on draw 5. A first run of this bench produced -0.89 and, with it, a headline: the best model on the held-out set is the worst under shift. One more data draw destroyed it.

Two readings of that column are both defensible from the numbers alone. One says the negative values in three draws out of five point at a real inverse relationship that the two positive draws are too noisy to show. The other says a quantity whose sign moves between draws is uninformative and the negatives are coincidence. Neither reading is settled by five draws, and the second is the one that survives contact with more of them so far. What is not defensible is quoting draw 1 on its own, which is how the first version of this bench nearly shipped. Session 9 takes the whole episode apart.

A standard pipeline run prints one accuracy. That number is a sample of size one from the set of models the pipeline can return, and the sampling was done by whatever seed happened to be in the config file. Reporting it as the model's accuracy is the same move as reporting one patient's blood pressure as the ward's.

The fix is not to report the best of twenty, which is session 5's mistake in a different costume. It is to report the range. "0.949 held out, twenty seeds inside 0.019" is a stronger sentence than "0.949", and "0.765 shifted, twenty seeds inside 0.051" is a much weaker one than "0.765", which is the point.