Where we start
Five sessions have given you Breiman's distinction, Box's standard for a useful model, Tukey's discipline of exploring before confirming, a bench on which both cultures scored within a few biopsies of each other, and Donoho's claim that a working data scientist uses both. What they have not given you is a rule for choosing, because the people who argued it most carefully never agreed on one. The honest version of a rule is a set of questions, each one somebody's objection to somebody else.
★ What you walk out with tonight. Each discussant's strongest point in his own words, three questions to ask before choosing a culture with the name of the person who would insist on each, the bench's results read as a trade-off frontier rather than a scoreboard, and a four-number scorecard on real biopsies that a person could check by hand.
Four replies, and nobody changed sides
7 min liveBreiman's own summary of the discussion: "Manny Parzen and Bruce Hoadley are more or less in agreement, Brad Efron has serious reservations and D. R. Cox is in downright disagreement." Each reply is worth reading for what it concedes as much as for what it attacks.
LiveCox: start from the question, and beware changed conditions3 min▶
Cox calls the paper's picture of mainstream statistics "a caricature" and adds that "like all good caricatures, it contains enough truth and exposes enough weaknesses to be thought-provoking". His first move is on the starting point: "Professor Breiman takes data as his starting point. I would prefer to start with an issue, a question or a scientific hypothesis."
He concedes, "although it goes somewhat against the grain to do so, that there are situations where a directly empirical approach is better", naming short term economic forecasting and real-time flood forecasting. Then comes the objection that has aged best: "Often the prediction is under quite different conditions from the data". His examples are the likely course of the v-CJD epidemic in the United Kingdom, and the effect on annual cancer incidence in the United States of cutting the medical use of X-rays by 10 percent. A model graded on held-out rows drawn from today's conditions says nothing about a world the data never contained. That question now has its own discipline: Causal Inference ↗.
He ends with an aphorism, "Better a rough answer to the right question than an exact answer to the wrong question", which he says is "due perhaps to Lord Kelvin". Tukey had printed a longer version in 1962, as session 3 showed: two attributions for one idea, neither origin confirmed.
LiveEfron: a hundred years of unbiasedness, and two rules2 min▶
Efron restates Breiman's point historically: the twentieth century was "100 years of unbiasedness", powerful but "purchased at a price", because it needs a high ratio of sample size to unknown parameters. The new algorithms, "black boxes with enormous numbers of adjustable parameters ('knobs to twiddle')", must be succeeding through some form of biased estimation, and there is little theory for that. Hence his two rules of faddish wandering: "Rule 1. New methods always look better than old ones." "Rule 2. Complicated methods are harder to criticize than simple ones."
He also disputes the 2 percent, says estimation and testing are themselves a form of prediction, and closes: "The whole point of science is to open up black boxes, understand their insides, and build better boxes for the purposes of mankind."
Self-studyHoadley and Parzen, and Breiman's rejoinder4 min▶
Hoadley describes Fair, Isaac's credit scoring since the 1960s as an algorithmic culture built by engineers: prediction characteristics, segments that turn out to be the leaves of a shallow tree, scorecards fitted to maximise separation, "no formal tests of fit and no tests of score weight statistical significance". He pushes back twice: test-sample performance "does not guarantee high performance on future samples, things do change", and accuracy need not cost interpretability, since regulators require "super interpretability". His "Ping-Pong theorem": given each other's best score and the data, Breiman's forest could beat his scorecard and his scorecard could then beat the forest.
Parzen renames the two goals "management" (profit, practical predictions) and "science" (truth, understanding), notes that Student's t-test was invented as a management tool at Guinness, and says "statistics has many cultures".
Breiman's rejoinder agrees with Cox that data models suit well-understood mechanisms and situations with too little data, then parts company: "Given a statistical problem, find a good solution". To Efron's rules: "Prove that you've got a better mousetrap and I'll buy it." To Hoadley: "In both stock prediction and credit rating, the priority is accuracy. Interpretability is a secondary goal that can be finessed." To Parzen: he keeps two cultures because the division "is pretty clear cut", and he puts Bayesians in the data modelling culture.
The X-ray question. Cox's example is worth running in your head. You have excellent records of cancer incidence and of medical X-ray use. A forest graded on held-out years would predict next year's incidence well, because next year looks like the data. Now the question is what happens if X-ray use falls by 10 percent, a world none of the rows came from. Accuracy on unseen rows from the old world is silent on it. That is the case where Cox's culture, a model of the process linked "with other sources of information", has no competitor, and Breiman agreed in his rejoinder: "I agree."
Three questions, three owners
5 min liveThe discussants disagree about which culture should be the default. They largely agree about which facts about a problem should decide it. Turn each one's strongest point into a question and ask them in order. The answer is a culture for this question, with the name of the person who would dissent.
LiveWhy the order matters3 min▶
Cox's question comes first because it can make the other two irrelevant. If the model will be used to predict the consequence of an intervention or under conditions the data never covered, held-out accuracy from the old conditions is not evidence, and a readable model built on the same rows is no better placed. Breiman agreed that this case needs "some understanding of the underlying process".
Hoadley's question comes second because it constrains the form without choosing the culture. A scorecard, a short tree or a regression on a handful of features can be read and still be graded the algorithmic way, on rows it never saw. On the session 4 bench the regression on the creators' three features did exactly that: four numbers, 167 of 171 held out.
Breiman's question comes last because it only decides between the remaining options. With rows to spare, hold some out and let the most accurate model win; his rejoinder concedes that with too little data "simple models can then be useful in giving qualitative understanding".
Self-studyWhat the tree deliberately leaves out2 min▶
It says nothing about which of several equally accurate models to believe, which is the Rashomon problem Breiman named and ML Epistemology, leader session 2 ↗ teaches. It says nothing about what a test score entitles you to claim beyond the rows it was measured on: leader session 1 ↗. And it takes no side between frequentist and Bayesian data models: Bayesian, leader session 5 ↗. A choice of culture is the first decision, not the last one.
Fraud control at 0.5 percent. Hoadley's comment describes a transaction fraud system where staff can review only 0.5 percent of transactions, so what the client wants is "a score that will maximize the dollars saved", not the lowest misclassification error. The culture question and the metric question are separate: an algorithmic model can still be optimising the wrong number. He called the challenge maximising "the generalization dollars saved, not generalization error".
A frontier, not a scoreboard
4 min liveDonoho's reading of the whole exchange, sixteen years later: "Each data scientist in practice uses tools and viewpoints from both of Leo Breiman's modeling cultures." The session 4 bench shows what that looks like when both numbers are plotted at once. No model is best on both. Some models are worse than another on both, and those are the only ones you can rule out.
LiveReading the frontier honestly2 min▶
A model is ruled out only when another model is at least as good on both numbers and better on one. The 30-feature forest (1,647 questions, 167) is ruled out by the 30-feature regression (31 numbers, 170). Between the two regressions there is a genuine choice: three more biopsies right, or 27 fewer numbers to read. Which one to ship is Hoadley's question from Part 2, not a statistical one.
Two cautions from earlier sessions. The forest's own seeds move it between 165 and 169, so the vertical positions of the violet dots are each a range of about four biopsies. And the creators documented that these classes are linearly separable using all 30 features, which is the best possible case for the data modelling culture. Breiman's microarray example, 81 samples and 4,682 genes, is the opposite case, and there he reports the forest's error "was low" with no variable deletion at all.
Self-studyThe Ping-Pong theorem as a working method2 min▶
Hoadley's "Ping-Pong theorem" is a joke with a method inside it. Reveal your best held-out score to the other culture and let them try to beat it; then take their score back. Each round, the target is a number both sides can see and neither can argue with. Donoho's common task framework, session 5, is the same game run by a referee. Played inside one team, it is the honest way to use both cultures: build the readable model, let the algorithmic one challenge it on the same untouched rows, and keep whichever the question in Part 2 says you can live with.
Trees fitted to explain the network. In his rejoinder Breiman tells of a machine learning researcher at a money management firm who used large neural nets for stock prediction. Asked how he explained them to clients: "Simple," he said; "We fit binary trees to the inputs and outputs of the neural nets and show the trees to the clients. Keeps them happy!" Breiman offers it as evidence that interpretability "can be finessed". Hoadley's regulators would call it the reason interpretability cannot be.
A scorecard a person can check, and six questions to place
★ 22 min · everyone buildsFirst the terracotta leaf of Part 2: the regression on the dataset creators' three features, printed as a formula a clinician could evaluate with a calculator, and graded on the same held-out 171. Then the three questions, applied to six real problems from the papers you have read. The outputs of the code are what it prints; the placements are yours to argue.
Fit the creators' three features on the session 1 split. Same seed, same 398 and 171, same penalty as the bench.
Print it as a scorecard. The means and spreads come from the training rows only, so the formula is exactly what the model computes.
Check it by hand. Pick any held-out biopsy, compute the score with a calculator from the three printed lines, and compare with lr.decision_function. They must agree. That agreement is what "a person can read it" means in practice; no forest offers it.
Place six real questions. Each comes from a paper you have read. For each, answer the three questions in order, name the culture, and name the discussant who would dissent and why.
Write the culture memo. One paragraph per question, in the same four lines every time. The dissent line is not optional: it is where the next reviewer will push.
Question 4 is the trap. Breiman's objection to the salary study was not that it used the wrong culture. It was that nobody first asked "Can the data gathered answer the question posed?" When the honest answer to that is no, neither culture helps, and the memo should say so.
After the last session
45 min- Read the four comments and the rejoinder (Statistical Science 16(3), pages 216 to 231). Underline one concession in each comment; they are easier to miss than the attacks.
- Write culture memos for two models at work, one you built and one you inherited. Use the four-line template. Show them to a colleague and ask them to argue the dissent line.
- Plot your own frontier. For one problem, fit a readable model and an algorithmic one on the same untouched rows, and plot held-out score against how much a person must read. Say which models are ruled out and which choice remains.
- Play one round of Ping-Pong. Give a colleague your best held-out score and your training data, and ask them to beat it with the other culture. Record both numbers and the size of the gap against your seed noise.
Three questions
Clear all three and session 6 gets stamped in your passport.
1 · What is the objection in Cox's comment that Breiman accepted in his rejoinder?
Breiman quoted the passage and answered "I agree." The disagreement that remained was about how often problems are like that, and whether data models should be the default elsewhere.
2 · Hoadley's "Ping-Pong theorem" says each culture could beat the other's revealed best score. What does that imply for practice?
Hoadley's own conclusion was that "all the competing models in this game of Ping-Pong would surely be algorithmic models. But some of them could be interpretable." Graded one way, readable the other: the terracotta leaf.
3 · After six sessions, which statement is the honest one?
The bench measured one nearly separable dataset; Breiman's examples and Cox's examples point in different directions because they are different problems. Donoho's resolution, use both, is a practice, not a verdict on who was right.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 6 cheat sheet · pin this
That is the whole argument: Breiman's two cultures, Box's useful models, Tukey's untouched confirmation, the bench, Donoho's fifty years and the replies that kept it honest. Where it goes from here, why to believe a model at all and what a benchmark can license, is the subject of ML Epistemology ↗. Back to the course home for your passport.