Equally good, differently wrong

The term comes from D'Amour et al. (2020), "Underspecification Presents Challenges for Credibility in Modern Machine Learning". Their definition is worth reading slowly, because every word in it is load-bearing: "An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain." Not "a badly built pipeline". Not "a pipeline with a bug". A pipeline that does its job, returns a strong model, and could just as easily have returned a different strong model.

The consequence they state is the part that belongs in a review: "Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains." Treated as equivalent is the phrase to sit with. Nobody decides to treat them as equivalent; it happens by default, because the only evidence on the table says they are.

This is not a quirk of one field. D'Amour et al. ran stress tests across computer vision, medical imaging, natural language processing, clinical risk prediction from electronic health records, and medical genomics. The paper carries forty authors, which is itself a signal of how much ground they had to cover to establish that the pattern was general rather than local.

• Pipeline. Everything that turns data into a fitted model: the features, the architecture, the training schedule, the stopping rule. Fixed here, deliberately.
• Seed. The arbitrary number that sets the initial weights and the order examples arrive in. The only thing that varies across the twenty models tonight.
• Underspecification. A pipeline that can return many models with equally strong held-out performance which behave differently once deployed.
• Spread. The distance between the best and worst of the twenty on one set. Held-out spread and shifted spread are measured on different sets, from the same twenty models.
• Stress test. Scoring the family on data chosen to break a specific assumption, rather than on more data of the same kind. It separates models that a held-out set cannot.
• Behavioural reliance. How much a model's answers move when one named feature is changed and nothing else is. Measured here by flipping the ward on every patient and counting changed predictions.
• Data draw. One fresh sample of patients from the same world. Repeating a finding across draws is how you learn whether the finding is about the world or about the sample. Session 6 returns to this.

Teams argue about architecture, features and regularisation, and those arguments leave a trail: a decision log, a slide, a comparison table. The seed leaves nothing. It is the number a library picks when nobody supplies one, and it is not usually in the model card, the review pack or the change request. The twenty models tonight differ by exactly that and nothing else.

Which makes the finding awkward to file. It is not a data problem, because the data are identical. It is not an architecture problem, because the architecture is identical. It is a property of the pipeline as a whole: the training data plus the design constrain the model a great deal, and then they stop constraining it, and something arbitrary finishes the job.

Accuracy is a summary, and two models can reach the same summary by different routes. A blunter instrument is to change one input and watch what happens. At Thornbury the test is to flip the ward on every patient, leave the two clinical signals alone, and count how many predictions change their answer.

Across the twenty seeds that number runs from 0.70 to 1.00. Some of these models change their mind about seven patients in ten when only the ward moves. Others change their mind about every single one. All of them scored about 0.949. If you had asked the review "how much does this model lean on the ward", the honest answer would have been "which of the twenty are you asking about", and nobody in the room would have had a way to tell.

The instinct on seeing a held-out spread of 0.019 is relief: the pipeline is stable. On draw 4 the held-out spread is 0.003, which is tighter still, and it is the worst case in the whole table. The shifted spread there is 0.034, eleven times the held-out spread. The tighter the agreement where you looked, the more confidently the family will be treated as one model, and the less the agreement was telling you.

This is the same shape as session 1's point about buying a larger test set. Precision on the quantity you can measure is cheap and reassuring. It is not evidence about the quantity you care about, and when it is high it actively suppresses the question.

In every draw the shifted spread is larger than the held-out spread, by between 2.7 and 11.3 times. Models that are indistinguishable where you looked are distinguishable where you did not. That sentence is the whole of underspecification stated as a measurement, and it is stable enough to build a policy on.

What it buys you in a review is a specific expectation. Whatever agreement the validation showed between candidate models, expect several times more disagreement once anything about the world changes. The held-out set is the setting in which the family looks most alike, by construction, because that is the set the pipeline was tuned against.

The correlation between a model's held-out accuracy and its shifted accuracy runs from -0.89 on draw 1 to +0.09 on draw 4. There is no pattern in that column. The honest reading is the flat one: held-out accuracy does not predict shifted accuracy, and it does not reliably anti-predict it either. It is close to uninformative about it.

The tempting sentence, and the one to refuse, is "the best model on the held-out set is the worst under shift". It is vivid, it sounds like wisdom, and one draw of this bench supported it beautifully before another draw destroyed it. Practitioner session 9 tells that story in full; tonight the point is only that the sentence is not available, and neither is its opposite.

Here is the version for a board paper. "We validated the model" does not identify which model was deployed. The validation identifies the pipeline and a performance band. Inside that band sits a family of models with behavioural reliance on a single feature ranging from 0.70 to 1.00, and the member of the family that shipped was chosen by an unrecorded number.

The direct operational consequence is about retraining. A retrain on refreshed data with a new seed returns a different member of the family, and it will pass the same validation with a similar number, because that is what the band means. So the change is invisible to the control that is supposed to catch changes. The paperwork carries over; the behaviour does not have to.

None of this makes retraining wrong. It makes "unchanged performance" the wrong evidence that nothing changed. What would be evidence is the behavioural measure: does the new model lean on the same features by the same amount as the one it replaces.

D'Amour et al. did not find this in one domain and generalise. They built stress tests in computer vision, medical imaging, natural language processing, clinical risk prediction from electronic health records, and medical genomics, and found the same shape in each: strong and equivalent held-out performance, divergent behaviour once the deployment conditions moved. Forty authors on one paper is what covering that much ground costs.

Cite it as D'Amour et al. A short author list invented for convenience will be wrong, and on a paper with this many contributors it is also unfair. Thornbury is a much smaller and fully controlled version of the same experiment, which is why every relationship in it can be checked against what the model actually learned.