All models are wrong, some are useful

Box's 1976 paper, "Science and Statistics", appeared in the Journal of the American Statistical Association, volume 71. Read at source, it does not contain the sentence on the mugs. It contains the idea, twice, on the same page, p.792, and each time the words "all models are wrong" are the first half of an argument about what to do next.

Section 2.3, Parsimony: "Since all models are wrong the scientist cannot obtain a 'correct' one by excessive elaboration." He adds that "overelaboration and overparameterization is often the mark of mediocrity". Section 2.4, Worrying Selectively: "Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad."

The exact sentence arrives three years later, as a section heading on p.202 of Box's chapter "Robustness in the Strategy of Scientific Model Building", in the 1979 volume Robustness in Statistics edited by R. L. Launer and G. N. Wilkinson. The chapter was also issued as a Wisconsin technical report in May 1979; that copy is reported, not read. Box and Draper's 1987 book Empirical Model-Building and Response Surfaces is widely quoted as saying "Essentially, all models are wrong, but some are useful" (reported: the book was not opened for these pages).

So when a colleague cites the sentence, the precise citation is Box 1979, p.202. When they want the argument behind it, send them to 1976, p.792: that is where the useful part, the tigers and the mice, is written out.

The 1976 paper is not about models in the abstract. Box illustrates it with R. A. Fisher's work at Rothamsted, and his claim is that good statistics comes from exactly that kind of close contact with a real scientific problem. He describes science as "a motivated iteration" between theory and practice and draws it as a feedback loop, which is where Part 2's figure starts.

Two lines carry the attitude. The statistician knows that "in nature there never was a normal distribution, there never was a straight line", and yet useful results follow from assuming both. And "we cannot know that any statistical technique we develop is useful unless we use it." For theory that has lost contact with practice he coins a word, "mathematistry".

Read beside Breiman, this is the data modelling culture at its most self-aware. Box never claims the equation in the box is nature's. He claims that a deliberately wrong equation, criticised against data again and again, is how science makes progress.

A synthesis, not a quotation. Neither Box nor Breiman wrote this list; it is built from their words so that "useful" becomes something you can test rather than assert.

1. It beats the base rate on rows it never saw. Breiman's perception (d): "Predictive accuracy on test sets is the criterion for how good the model is." A model that cannot beat the do-nothing rule on unseen rows is not useful for anything, whatever its coefficients say. This is the algorithmic culture's check, and a data modeller loses nothing by running it.
2. It is wrong only where the decision does not care. Box's "importantly wrong": every model is wrong somewhere, so the question is whether the wrongness sits under the decision you are about to make. A tiger for one use can be a mouse for another; Part 3 shows the same coefficient being both.
3. Its structure tells you something. Box's "illuminating". The ideal gas law earns its place because its structure is informative, not because it is exact. A model whose structure you cannot read, or whose structure you should not read, can still pass checks 1 and 2, but it does not pass this one.

Notice what is missing: a high R squared on the training rows, every coefficient significant, a clean goodness-of-fit test. None of those is on the list, and Part 3's real world story shows a model failing nearly all of them and still winning.

D. R. Cox, replying to Breiman in the same issue of Statistical Science, states the data modelling ideal in one sentence: "Descriptively appealing and transparent methods with a firm model base are the ideal." That is check 3 raised to a principle.

He is also honest about its cost. The significance tests and confidence intervals the culture reports "are often lower bounds to real uncertainty", because they are computed as if the model were right. Box's line and Cox's line are the same warning from two directions: the model is wrong, so the uncertainty it reports about itself is too small.

Fourteen years before Box's paper, John Tukey quoted his colleague Martin Wilk in "The Future of Data Analysis" (p.7): "The hallmark of good science is that it uses models and 'theory' but never believes them." It is the shortest statement of the whole attitude, and session 3 reads the rest of Tukey's paper.

One fault line this session leaves alone: frequentist versus Bayesian runs inside the data modelling culture, not between the cultures. Breiman: "I would include Bayesians in the data modeling culture." That argument lives in Bayesian, leader session 5 ↗.

For prediction, a mouse. The model is 170 of 171 on biopsies it never saw, well above the 62.6 percent base rate, so it passes check 1. Nobody deciding "send this biopsy for a second read" is harmed by how the weight is split between radius, perimeter and area, because the prediction uses them all together.

For reading mechanism, a tiger. Read mean compactness off the model and you would report that compact nuclei lean benign, +0.57. On the training rows the feature's correlation with benign is -0.59: more compact nuclei are more often malignant. The coefficient is not describing compactness; it is correcting its neighbours. The model fails check 3 for that sentence, and check 2 for any decision built on it.

This is Box's "importantly wrong" made concrete: the error is the same seven numbers either way. What changes is the decision sitting on top of them. The honest report says both halves: "useful for prediction; do not read these seven coefficients".

Radius, perimeter and area are three ways of measuring one thing. On the training rows mean radius and mean perimeter correlate at 0.998, mean radius and mean area at 0.990, worst radius and worst perimeter at 0.993, worst radius and worst area at 0.987. Across all 30 features, 21 pairs correlate above 0.9 in absolute value.

When features are near-duplicates, the fit can trade weight between them almost freely, and a feature can take whatever sign best corrects the others rather than one that describes itself. Alone, worst area carries a coefficient of -4.41; in the full model it carries -0.85, and the rest of the size signal is spread over five siblings. With Box's parsimony in mind, the fix is not a cleverer reading of 31 numbers. It is fewer features: the build-along refits on the dataset creators' 3 and gets 4 numbers, 0 sign surprises and 167 of 171.

The broader fact that many equally good models can tell different stories about the same data is the Rashomon effect, and it has its own session: ML Epistemology, leader session 2 ↗.