Reading a benchmark

Every benchmark returns a number about a specific quantity, under specific conditions, using a specific rule for what counts as wrong. Construct validity asks whether that quantity is the one you care about. It is a different question from whether the reading is accurate, and it is the one that usually goes unasked, because an accurate reading of the wrong quantity looks exactly like an accurate reading.

The practical form of the question splits into three, and every leaderboard row answers all three silently. What task was formalised. Whose data it was computed on. How predictions were converted into one number. A rank is a claim about a method paired with those three things, and the pairing is what does not come with you when you copy the number into your own document.

• Construct validity. Whether the quantity an instrument measures is the quantity you intended to measure. Precision and reproducibility say nothing about it.
• Task. The decision the benchmark formalises: what is predicted, from what inputs, at which moment.
• Distribution. Whose data, collected when, by what process, and who is not in it.
• Scoring rule. How a set of predictions becomes one number, which is also the decision about which direction of error is allowed to matter.
• Task set. The collection of benchmarks a community happens to use. It is a choice, and it participates in the result.
• Level and ordering. How well a method scores, against whether it scores better than another one. They do not travel equally well, and most citations carry the wrong one.
• The licence. The written statement of what a published number supports for your problem. Three sentences, attached to the citation.

Published benchmark numbers are usually computed correctly. The code is often released, the protocol is documented, and a second team running it gets the same figure. All of that is evidence about precision and none of it is evidence about which quantity was measured. A thermometer that reads to two decimal places is still the wrong instrument for measuring a room's humidity, and nobody catches that by checking the decimal places.

The failure shows up in a particular way: the team that cites the number is not wrong about the number, so the conversation never lands on anything checkable. Naming the three parts converts it into something checkable. Once the benchmark's task is written as one sentence next to yours, disagreement becomes possible, and disagreement that can be settled is worth far more than agreement that cannot.

After the three, there is a fourth that nobody else can answer: what would this method score on our data. It is the only question whose answer settles an absolute claim, and the reason to ask it last is that the first three tell you how large the gap you are trying to close is likely to be. A benchmark whose task, distribution and scoring rule are all close to yours needs a small confirmation. One that differs on all three needs a measurement, not a confirmation.

Asking it also changes what a supplier conversation is about. A rank is not falsifiable by you. A figure computed on a set you kept clean is, which turns a sales claim into a testable one at the cost of some calendar time and no statistics at all.

It does not mean the results are random, and it does not mean anybody gamed anything. It means the set of tasks a community settled on is itself a choice, made before the methods existed and usually for reasons of history rather than design. That choice is part of the answer. A method that suits the tasks in the set will place well in it, and placing well in it is what the word leading means in practice.

The practitioner consequence is a counting habit rather than a rule. One benchmark is one instrument reading. If a lead exists on one task set, the claim has survived nothing yet. If it holds across several unlike task sets, it has survived something and the ordering is worth carrying into a shortlist. If the lead exists on exactly the benchmark the authors are best known for, say that out loud when you cite it.

A shared benchmark is how a field makes progress comparable at all. Without one, every claim is a demonstration and nothing can be ranked against anything. Reading the lottery as a reason to discard published evidence replaces a calibrated inference with no inference, which is a worse position to argue from and much harder to correct.

The useful reading is about the strength of an inference rather than its existence. A rank is real evidence of modest strength about a comparison, and the strength rises with the number of unlike task sets the comparison survives. That is a dial, not a switch, and treating it as a switch is how teams end up choosing models on vibes after deciding the numbers cannot be trusted.

The intuitive explanation for a fresh test set producing worse numbers is that the old one had been scored against so often that the community had quietly fitted to it. Recht, Roelofs, Schmidt and Shankar concluded otherwise for these drops: the cause was the models' limited ability to generalise to slightly harder images rather than contamination through repeated use. Roelofs, Shankar and colleagues went at the same suspicion from another angle, across over one hundred Kaggle competitions where entrants score repeatedly against a public holdout and a separate set decides the final ranking, and found little evidence of substantial adaptive overfitting.

Against that, session 5 measured the mechanism directly at Thornbury and it is real: optimism of 0.0027 at one candidate rising to 0.0072 at fifty, then flat. Small, and not zero. Three findings, all sound, and the temptation is to collapse them into a single slogan in one direction or the other. Keeping them apart is the skill. Reuse is a real mechanism with a modest measured cost. The distribution gap is a separate and larger effect. A published number can be hurt by both and they need different responses.

Suppose you rebuild a benchmark's test set yourself, following its collection process, and your models score several points lower. That fall is consistent with two very different stories: the community fitted to the old set, or your rebuild is slightly harder than the original in ways nobody intended. Recht, Roelofs, Schmidt and Shankar ran into exactly that ambiguity and landed on the second explanation.

The observation that separates them is the ordering. If the community had fitted to the old set, the methods that had been tuned hardest against it would fall furthest and the ranking would scramble. If the new set is simply harder, everything falls together and the ranking survives. The ranking surviving, plus gains on the original sets translating to larger gains on the new ones, is what made the second story the better fit. That is a diagnostic you can run yourself whenever you build a second test set.