Where tonight sits
Session 1 laid out three links that carry a score from a test set to a decision: the same world, an untouched draw, and the right target. Session 2 was underspecification and session 3 was what happens once a number becomes a target. Tonight is the third link at the scale of a whole field: benchmarks, and what a published rank entitles you to expect on your own problem. Session 5 takes the same question one level up again, to a body of published evidence that was wrong in the same way nearly three hundred times. The running example is still Thornbury, the hospital group predicting which patients deteriorate within 24 hours, although tonight most of the evidence comes from outside it.
★ What you walk out with tonight. A replacement for the question "what did it score". Instead: what task, whose distribution, what scoring rule, and what would it score on ours. You also walk out with two pieces of evidence you can cite by name: the benchmark lottery, which shows a ranking can move when only the task set moves, and the ImageNet and CIFAR-10 result where genuinely new test sets cost 3 to 15 accuracy points and the ordering of the models largely survived anyway.
LiveA benchmark is an instrument, so ask what it measures4 min▶
Treat a benchmark the way you would treat any instrument somebody wheels into the room. It has a range, it has conditions it works under, and it returns a number about a specific quantity. The engineering word for the question that matters is construct validity: is the thing the instrument measures the thing you care about? Not "is the reading accurate", which is usually yes, but "is it a reading of my quantity", which is usually unexamined.
A benchmark score has three parts hiding inside it. There is a task, which is a decision someone chose to formalise in a particular way. There is a distribution, which is whose data, collected when, by what process. And there is a scoring rule, which decides what counts as being wrong and by how much. A rank is a claim about the pairing of a method with those three things. It is not a property of the method that travels with it.
A vendor shows Thornbury a model ranked first on a public clinical prediction benchmark. Every part of that claim can be true while none of it transfers: the benchmark's task may be a 30-day outcome where Thornbury needs 24 hours, its distribution may be a teaching hospital's intake rather than a district group's, and its scoring rule may treat the two directions of error as equally costly when Thornbury's ward staff emphatically do not.
Self-studyThe words tonight keeps using4 min▶
- Construct validity. Whether the quantity an instrument measures is the quantity you intended to measure. A score can be precise, reproducible and entirely about something else.
- Task. The decision a benchmark formalises: what is predicted, from what, at what moment.
- Distribution. Whose data the benchmark is built from, collected when and by what process.
- Scoring rule. How the benchmark converts a set of predictions into one number, which decides what kind of error is allowed to matter.
- Task set. The collection of benchmarks a community happens to have settled on. Change the collection and you can change who wins.
- Distribution gap. The distance between the data a number was computed on and the data a model will meet, even when nobody did anything wrong.
- Level and ordering. Two different things a benchmark can be evidence for. How well a method does, and whether it does better than another. They are not equally fragile.
Change the task set, change the winner
10 min liveDehghani, Tay, Gritsenko, Zhao, Houlsby, Diaz, Metzler and Vinyals put a name to something a lot of practitioners had felt and nobody had stated cleanly. Their 2021 paper, arXiv 2107.07002, calls it the benchmark lottery, and it "postulates that many factors, other than fundamental algorithmic superiority, may lead to a method being perceived as superior". They go on to show that "the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks". They argue the fragility reaches across natural language processing, computer vision, information retrieval, recommender systems and reinforcement learning, which is to say it is not one community's local problem.
LiveNobody has to cheat for the lottery to operate4 min▶
The word lottery does a lot of work here and it is worth being careful with it. It does not mean the results are random or that anyone has gamed anything. It means that the set of tasks a community happens to have adopted is itself a choice, usually a historical accident, and that choice participates in the result. Whoever picked the task set picked part of the answer, long before any method was trained.
For a leader this reframes what a rank is worth in an argument. Two teams citing different benchmarks to support opposite conclusions are not necessarily disagreeing about facts. They may both be reporting correctly, from different instruments, about different quantities. The useful move is not to litigate which leaderboard is the real one. It is to ask what each one measures and which one is closer to the decision you are about to make.
Self-studyWhy benchmarks still earn their keep3 min▶
None of this is an argument for ignoring benchmarks, and a leader who takes it that way has taken the wrong thing. A shared benchmark is how a field makes progress legible at all: without one, every claim is a demo and nothing can be compared. The lottery result is a caution about the strength of the inference, not a reason to stop measuring.
The healthy version is a habit rather than a rule. Treat one benchmark as one instrument reading. Ask for a second one before you believe an ordering, and ask for a third before you believe an ordering is large. When a method leads across several unlike task sets, the claim has survived something. When it leads on exactly the benchmark its authors are best known for, you have learned considerably less.
What happens when somebody builds a fresh test set
8 min liveRecht, Roelofs, Schmidt and Shankar asked a question in 2019 that sounds almost too simple: "Do ImageNet Classifiers Generalize to ImageNet?" They built genuinely new test sets, following the original collection process as closely as they could, and scored the existing models on them. Accuracy fell by 3 to 15 percent on CIFAR-10 and 11 to 14 percent on ImageNet. That is the half of the result everyone remembers. The other half matters just as much, and tonight both halves get said in the same breath.
LiveTwo findings that people insist on separating4 min▶
The first half, the drop, is quoted constantly as proof that published numbers are inflated. The second half, that the ranking largely survived, is quoted much less because it is less alarming. Reporting either one alone gives a false picture, and a leader who only carries the first will make bad decisions in a predictable direction: refusing to use comparative evidence at all, which is worse than using it carefully.
Put together they say something quite specific. A benchmark number is decent evidence about which method is better and poor evidence about how well any of them will do on your data. The ordering is the part that transferred; the level is the part that did not. That asymmetry is not a technicality, it is exactly the shape of what a benchmark can license, and most vendor conversations get it backwards by leading with the absolute figure.
Thornbury is choosing between two vendor models, both citing the same public benchmark, one two points ahead of the other. The defensible reading is that the leading model is probably the better of the two. The indefensible reading is the one on the slide: that either of them will hit the benchmark figure on Thornbury's patients. Nothing in that number was computed on a district hospital group's intake.
LiveNote what the authors ruled out3 min▶
The obvious story about a fresh test set producing worse numbers is that the old test set had been overused, scored against so many times that the community had quietly fitted to it. Recht, Roelofs, Schmidt and Shankar concluded otherwise: the drops came from the models' limited ability to generalise to slightly harder images, rather than from test-set contamination through repeated use.
That conclusion should sit awkwardly next to session 3, and it is meant to. Session 3 measured a real cost to selecting against a set, found it under one accuracy point and saturating. This result says the much larger fall on new data came from somewhere else entirely. Both can be true and both are: reuse is a real mechanism with a small measured cost, and the distribution gap is a separate and larger effect. Holding the two apart is most of what it takes to read this literature honestly.
Self-studyWhy "what would it score on ours" is the whole question3 min▶
The practical consequence is worth stating flatly. A benchmark number is evidence about relative method quality under one measurement regime, and close to no evidence about the absolute performance you will see. Asking "what did it score" is a weaker question than asking "what task, whose distribution, and what would it score on ours", and the second question costs nothing except the willingness to look less impressed for a minute.
It also has a clean procurement consequence. If the absolute figure does not transfer, then the only number that can settle an absolute question is one computed on your data, by your team, on a set you kept clean. A vendor who will not agree to that is telling you something. A vendor who will has just converted an unfalsifiable claim into a falsifiable one, which is where session 6 picks up.
Take a published number apart in front of everyone
20 min build-alongNo laptop. Bring the most impressive external number anyone in the room has been shown recently: a vendor slide, a paper cited in a strategy document, an internal team quoting a public leaderboard. Work through it out loud. The aim is not to discredit it. The aim is to convert it from a verdict into evidence of a known size.
Find the instrument. Which benchmark, by name, and who built it. If the slide cannot say, you are looking at a claim with no instrument behind it and the exercise is already over.
State the task in one sentence, as a job. What is predicted, from what inputs, at what moment. Then say your own task the same way and put the two sentences side by side. Most of the gap is visible right there.
Name the distribution. Whose data, collected when, by what process, and who is in it that is not in yours. "Patients from a teaching hospital in 2019" and "our intake now" are different populations even when every column matches.
Ask what the scoring rule forgives. Which direction of error does it treat as cheap? Who pays for that error in your setting? This is session 1's third link, the right target, arriving at the scale of a whole field.
Ask for a second benchmark. Does the ordering hold on an unlike task set? One is a reading, two is a claim, three is a pattern. If the lead exists on exactly one leaderboard, say so out loud and let the room hear it.
Convert the rank into a testable question. "You are probably the better of the two" is what the evidence supports. Write down what it would take to learn the absolute number on your own data, and who would run it.
The one-sentence version to take into a meeting. "That rank tells me you are likely better than the alternative on that benchmark's task and distribution; it does not tell me what you will score on ours, so what would it take to find that out?" It is not a hostile question. It is the only one that turns a leaderboard position into something a decision can rest on.
Three questions
A vendor's model is ranked first on a well-known public benchmark. What does that rank entitle you to expect on your own problem?
The ordering is the part that tends to transfer and the level is the part that does not. Recht, Roelofs, Schmidt and Shankar measured exactly this shape: accuracy fell by 3 to 15 points on CIFAR-10 and 11 to 14 on ImageNet against genuinely new test sets, while gains on the original sets still translated to larger gains on the new ones.
Recht, Roelofs, Schmidt and Shankar built new ImageNet and CIFAR-10 test sets and accuracy dropped. What did they conclude the cause was?
Contamination through repeated use is the intuitive story and the authors ruled it out as the explanation for these drops. That keeps it separate from session 3's finding, where selecting against a set does carry a real cost, measured under one accuracy point and saturating. Two mechanisms, different sizes, both real.
Two internal teams cite different benchmarks and reach opposite conclusions about which method is better. What does the benchmark lottery say about that?
The lottery postulates that factors other than fundamental algorithmic superiority can make a method be perceived as superior, and nobody has to act in bad faith for that to happen. Averaging unlike instruments does not fix it either; the question is which instrument measures the quantity your decision needs.
Before session 5
30 minFind one published result your area currently relies on that nobody in your team produced. A paper in a strategy document, a vendor's cited study, an industry figure that gets repeated. Write down two things about it: what data it was computed on, and how the data were split between what the model learned from and what it was scored on. If you cannot find out, note that too. Session 5 is about what a reader should do with evidence they did not produce, and that missing answer is the starting point.