learn-two-cultures-with-phoebe / Session 1 of 6
Learn The Two Cultures with Phoebe · Session 1 of 6

Two ways to ask the data

In 2001 Leo Breiman, a Berkeley statistician who had spent thirteen years as a consultant, told his own profession that it had built almost everything on one habit: assume an equation generated the data, estimate its parameters, and read the answer off the equation. He called that the data modelling culture, guessed it held 98 percent of statisticians, and set beside it a second culture that leaves nature's box shut and judges a model only by how well it predicts rows it has never seen. Four eminent statisticians replied in the same issue, and one of them disagreed with the main thrust outright. Tonight we read the argument from the page, then fit both cultures on the same 569 real biopsies and see what each one lets you say.

🟠 Advanced Leaders and practitioners Python, scikit-learn 45 min
0-3 · Welcome 3-18 · The black box and its two cultures 18-40 · Build-along: both cultures, same biopsies 40-45 · Q&A
Part 0

Where we start

Every argument you have heard about "explainable versus accurate", "statistics versus machine learning" or "we need a model the regulator can read" is a descendant of one 33-page paper and its discussion. Six sessions read that argument and the work of three other authors who shaped the field, each at source, and test the claims on real rows instead of taking a side. Two colours run through every figure and every widget: ochre for the data modelling culture, violet for the algorithmic culture.

🎫 Clear all three quiz questions to stamp your passport Live = presented in the 45 minutes Self-study = the full depth, after class ★ Build-along = run the code yourself

★ What you walk out with tonight. Breiman's distinction stated in his own words, precisely enough to classify any model you meet; the three consulting lessons that turned a probabilist into a member of the 2 percent; the hepatitis example that shows why an unchecked model's "important variables" can be fiction; and a script that fits both cultures on the same real biopsies and prints the two numbers each culture is judged by.

Part 1 · covers Breiman's black box and the two cultures

Nature's box, and two ways to fill it

6 min live

Breiman opens with a picture: a vector of inputs x goes into a black box, a response y comes out, and inside the box "nature functions to associate the predictor variables with the response variables". There are two goals, he says: prediction, and information about how nature makes the association. The cultures differ in what they put inside the box.

ONE BLACK BOX, TWO CULTURES drawn as Breiman draws it: y comes out on the left, x goes in on the right y x y x y x nature shut, and stays shut linear regression logistic regression Cox model unknown trees, neural nets WHAT IS REALLY THERE DATA MODELLING ALGORITHMIC nobody sees inside both cultures agree on that validation: yes or no goodness-of-fit, residuals validation: a number accuracy on unseen rows 98% of all statisticians, his estimate 2% of statisticians, many in other fields The disagreement is not about the box, which nobody can open. It is about what counts as checking your guess.
🔍 Click to zoom - the bottom row is the whole argument: a yes or no from a fit test, or a measured score on rows the model never saw
LiveThe two definitions, word for word4 min▶

The data modelling culture "starts with assuming a stochastic data model for the inside of the black box", for example that responses are generated by independent draws from response = f(predictors, random noise, parameters). The parameters are estimated from the data and the model is then used for information, prediction or both. Validation is "yes-no using goodness-of-fit tests and residual examination".

The algorithmic modelling culture "considers the inside of the box complex and unknown". Its approach is to find a function f(x), "an algorithm that operates on x to predict the responses y". Validation is "measured by predictive accuracy".

Notice what the distinction is not. It is not linear versus nonlinear, nor old versus new, nor statistics versus computer science. A logistic regression can be used in either culture: as a claim about how nature works, read off its coefficients, or as a predictor whose only credential is its score on held-out rows. Breiman's own test, from his reply to the discussants, is a single question: "are you modeling the inside of the box or not?"

The 98 and 2 percent are Breiman's estimate, printed in the figure as "Estimated culture population". Brad Efron replied in the same issue that the prediction culture, "at least around Stanford, is a lot bigger than 2%". Keep the numbers as rhetoric, not as a census.

Self-studyWhich culture are Bayesians in?2 min▶

Breiman answered this one himself: "I would include Bayesians in the data modeling culture." A prior and a likelihood are still a claim about what is inside the box. The frequentist versus Bayesian argument is a different fault line that runs through the data modelling culture, not between the two cultures, and it has its own course: Bayesian, leader session 5 ↗.

Real world

"What's the model for the data?" In the late 1970s a prominent Berkeley statistician visited Breiman in Los Angeles. Breiman described the decision tree method he had been building for clients. His friend's first question was the one above. Breiman tells the story as his first tip about what university research was like, and when he returned to Berkeley in 1980 he found that virtually every article in the flagship journals began the same way: "Assume that the data are generated by the following model".

Part 2 · covers the consulting projects and five perceptions

How a probabilist ended up in the 2 percent

5 min live

Breiman spent seven years as an academic probabilist, resigned, and spent thirteen years as a free-lance consultant before joining Berkeley's statistics department. He says his views came from the problems clients paid him to solve. Two of them are in the paper in detail, and both are built around the same mechanism: set rows aside, and let them grade the model.

THE OZONE PROJECT, EPA, MID-1970S predict tomorrow's ozone in the Los Angeles Basin from today's weather years 1 to 5: train 450+ weather variables, daily and hourly years 6 and 7: test The alert warnings would help more if issued 12 hours ahead; the test years say whether they can be. THE CHLORINE PROJECT, EPA does this compound contain chlorine? read it from its mass spectrum 30,000 compounds known structure and mass spectrum 25,000 train 5,000 test a tree of 1,500 designed yes or no questions 95% on both classes discriminant analysis was tried first; the variable length of a spectrum defeated it Neither project asked what equation nature uses. Both asked one question: how often is it right on rows it never saw?
🔍 Click to zoom - the terracotta block in each project is the set-aside rows, and it is the only part that grades the model
LiveFive perceptions he took back to the university3 min▶

Breiman lists them as the views he held when he left consulting, and they are worth reading slowly because four of them are uncontroversial and one is the whole fight:

  1. (a) Focus on finding a good solution, "that's what consultants get paid for".
  2. (b) "Live with the data before you plunge into modeling." Session 3 is about the man who made this a discipline.
  3. (c) Search for a model that gives a good solution, "either algorithmic or data".
  4. (d) "Predictive accuracy on test sets is the criterion for how good the model is."
  5. (e) Computers are an indispensable partner.

Item (c) is the one most people forget: he does not say "use algorithms". He says search both. Item (d) is the one the discussants fought about. D. R. Cox's reply accepts prediction as a test of a theory and then asks what happens when the prediction you need is "under quite different conditions from the data". Session 6 lets that argument run.

Self-studyThe court delay tree, and the judge who read it2 min▶

In the late 1970s Breiman studied the causes of delay in criminal trials for the Center for State Courts, using Colorado's computerised court data. The response was the time from arraignment to sentencing. He grew a large decision tree and explained it on an overhead projector to the assembled Colorado judges. One split was on District N, which had longer delays than the others. He said nothing about it. Walking out, he heard one judge say to another: "I knew those guys in District N were dragging their feet."

He uses the story to give single trees "an A+" on interpretability and "a B on prediction". Forests reverse the grades, which is the Occam dilemma of Part 3's self-study card.

Part 3 · covers what goes wrong when the model becomes the fact

The conclusions are about the model

4 min live

Breiman's sharpest sentence is a pair of truisms: "The conclusions are about the model's mechanism, and not about nature's mechanism", and so "if the model is a poor emulation of nature, the conclusions may be wrong." His worked example is a hepatitis dataset of 155 patients, where the measured accuracy of each culture decides whose list of "important variables" deserves any faith.

HEPATITIS, 155 PATIENTS, 19 COVARIATES test-set error, 100 runs with a random 10 percent held out · shorter is better everyone survives logistic regression random forest 20.6% 17.4% 12.3% 3.2 points 32 deaths, 123 survivors: the do-nothing rule already scores 79.4 percent Logistic, by coefficient size: variables 7 and 11 matter; using only those two, error rises to 22.9% Forest, by shuffling one at a time: 12 and 17, each alone 14.3%, and together no better, surrogates A model barely better than the do-nothing rule has not earned the right to name the important variables.
🔍 Click to zoom - the terracotta gap is everything the logistic model learned beyond "everyone survives", and its variable list rests on it
LiveWhy a fit test cannot rescue the model2 min▶

The data modelling culture checks its models with goodness-of-fit tests and residual plots, so the obvious reply is "check harder". Breiman's answer comes from three pieces of evidence. He once simulated a seven-dimensional regression with a controlled amount of nonlinearity, and standard goodness-of-fit tests "did not reject linearity until the nonlinearity was extreme". He cites Bickel, Ritov and Stoker: omnibus tests have little power unless the direction of the alternative is specified. And he reports William Cleveland, "one of the fathers of residual analysis", admitting at a 1993 Berkeley seminar that residual analysis could not uncover lack of fit in more than four to five dimensions.

His conclusion: "An acceptable residual plot does not imply that the model is a good fit to the data." A yes from a weak test is not evidence, and the measured error on held-out rows is the check he proposes instead. The same passage names what happens when many models pass: several equally good models give different pictures of nature. That is the Rashomon effect, and it has a session of its own in ML Epistemology, leader session 2 ↗.

Self-studyThe Occam dilemma, and how Breiman dissolves it3 min▶

Trees read well and predict moderately; forests of 50 or 100 trees predict very well and cannot be read. "So on interpretability, they rate an F." Breiman calls this the Occam dilemma, then argues it is the wrong question: "The goal is not interpretability, but accurate information." His claim, from the hepatitis and two other medical examples, is that a more accurate model gives more reliable information about which inputs matter, measured by shuffling one input at a time and counting the damage.

The claim is contested inside the same issue. Bruce Hoadley, from the credit-scoring company Fair, Isaac, replies that his segmented scorecards are "very accurate and very interpretable", and that regulators require it. Efron replies that "the whole point of science is to open up black boxes". Session 4 tests both claims on real biopsies; session 6 lets the disagreement stand.

Real world

The salary study "accepted as gospel". Breiman recalls a study, several decades before his paper, of gender discrimination in a university's faculty salaries: salary as the response, 25 variables describing academic performance, gender as a binary predictor. The gender coefficient was significant at 5 percent, and that was taken as strong evidence of discrimination. His objection was not to the conclusion but to the order of questions: "Can the data gathered answer the question posed? Is inference justified when your sample is the entire population? Should a data model be used?" The focus, he writes, "was on the model and not on the problem".

Build-along

Both cultures, the same 569 biopsies

★ 22 min · everyone builds

The data is the UCI Breast Cancer Wisconsin (Diagnostic) set: 569 fine needle aspirate biopsies, 30 measurements of the cell nuclei in each digitized image, and a diagnosis of malignant or benign. scikit-learn ships it as load_breast_cancer, which is byte-for-byte the UCI file. Seven steps fit one model from each culture on the same fixed split and print the two numbers each culture is judged by. Every output below is what the code prints.

ONE SPLIT, TWO CULTURES, ONE GRADER 569 biopsies 212 malignant 357 benign 398 to fit 171 held out split once, seed 2001 logistic regression 31 readable numbers forest of 100 trees 1,673 yes/no questions graded on the 171 regression 170 forest 167 base rate 62.6% The held-out rows are touched once, at the end. Everything upstream sees only the 398.
🔍 Click to zoom - the terracotta path from the split to the grader is the one that must never pass through either model

Load the biopsies and look at the classes. 30 numeric measurements per biopsy; the label is 1 for benign and 0 for malignant, scikit-learn's convention.

★ Step 1 · loadimport numpy as np from sklearn.datasets import load_breast_cancer data = load_breast_cancer() X, y = data.data, data.target # y: 1 = benign, 0 = malignant print(X.shape, np.bincount(y)) # (569, 30) [212 357]

Split once, stratified, and never again. Thirty percent of each class is set aside with a fixed seed. These are the same 171 rows the session 4 bench grades on.

★ Step 2 · the held-out rowsrng = np.random.default_rng(2001) test = [] for cls in (0, 1): idx = np.flatnonzero(y == cls) test += list(rng.permutation(idx)[: round(0.30 * len(idx))]) test = np.array(sorted(test)) train = np.setdiff1d(np.arange(len(y)), test) print(len(train), len(test)) # 398 171

Write down the line both cultures must clear. Calling every biopsy benign is a model too. Any score has to be read against it.

★ Step 3 · the base ratebase = max(y[test].mean(), 1 - y[test].mean()) print(f"base rate {base:.3f}") # 0.626

The data modelling culture: a logistic regression. Standardise on the training rows only, fit, then grade on the held-out rows. C=1.0 is scikit-learn's default penalty; the tight tolerance makes the coefficients match the bench to two decimals.

★ Step 4 · ochrefrom sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression sc = StandardScaler().fit(X[train]) lr = LogisticRegression(C=1.0, tol=1e-10, max_iter=10000) lr.fit(sc.transform(X[train]), y[train]) lr_pred = lr.predict(sc.transform(X[test])) print("logistic", (lr_pred == y[test]).sum(), "of", len(test)) # logistic 170 of 171

The algorithmic culture: a random forest. One hundred trees, each grown on a bootstrap of the training rows with a random subset of features tried at every split. How that works is taught in Ensemble Methods, session 2 ↗; tonight only its score and its size matter.

★ Step 5 · violetfrom sklearn.ensemble import RandomForestClassifier rf = RandomForestClassifier(n_estimators=100, random_state=0) rf.fit(X[train], y[train]) rf_pred = rf.predict(X[test]) print("forest ", (rf_pred == y[test]).sum(), "of", len(test)) # forest 167 of 171

The second number: what each model lets you say. The regression hands you one signed number per feature. The forest hands you a count of questions nobody will read.

★ Step 6 · the explainability readoutcoef = sorted(zip(lr.coef_[0], data.feature_names), key=lambda t: -abs(t[0]))[:3] for w, name in coef: print(f"{name:22s} {w:+.2f}") # worst texture -1.33 # radius error -1.08 # worst symmetry -1.07 nodes = sum(t.tree_.node_count - t.tree_.n_leaves for t in rf.estimators_) print("forest yes/no questions:", nodes) # forest yes/no questions: 1673

Where exactly do the cultures differ? Count the biopsies on which the two models give different answers, then re-run the forest with five seeds. The forest's own seed moves its score by about as much as the gap between the cultures.

★ Step 7 · the disagreementprint("disagree:", (lr_pred != rf_pred).sum()) # disagree: 3 for s in range(5): r = RandomForestClassifier(n_estimators=100, random_state=s).fit(X[train], y[train]) print(s, (r.predict(X[test]) == y[test]).sum()) # 0 167 · 1 167 · 2 168 · 3 168 · 4 168

Read the sign before the size. A negative coefficient here leans towards malignant, because the label codes benign as 1. The regression's largest coefficient is on worst texture, not on any of the size measurements a pathologist would name first. Session 2 finds out why, and it is not because texture matters most.

Homework

Before session 2

30 min
  1. Read Breiman's sections 1 to 5 (about eight pages of Statistical Science 16(3)). Mark every sentence where he is describing a practice and every sentence where he is describing a person. The paper is sharper on practices.
  2. Classify three models you use at work with Breiman's one question: is anyone modelling the inside of the box? Write one sentence each on how the model is validated.
  3. Run the build-along with random_state from 0 to 19 and write down the forest's lowest and highest score. Keep the range; session 4 needs it.
  4. Print all 30 regression coefficients next to each feature's correlation with the label. Count how many point the opposite way. Bring the count.
Check yourself

Three questions

Clear all three and session 1 gets stamped in your passport.

1 · A team fits a logistic regression, never reads its coefficients, and judges it only by accuracy on a held-out month. Which culture are they in?

The cultures are defined by what you claim about the box and how you validate, not by the algorithm's name. The same logistic regression is a data model when its coefficients are read as nature's mechanism and checked with fit tests, and an algorithmic model when it is just a predictor graded on unseen rows.

2 · On the hepatitis data, why does Breiman distrust the logistic regression's list of important variables?

The argument runs through measured accuracy: "Weak predictive accuracy can lead to questionable conclusions." Significance was never the problem; the model's weak grip on the data was, and nobody would have noticed without a predictive check.

3 · In the build-along the regression gets 170 of 171 and the forest 167. What is the honest reading?

One dataset settles nothing about a culture. Breiman himself says "I am not against data models per se", and the best model "could be an algorithmic model, or maybe a data model, or maybe a combination". Session 4 moves the levers and watches the ordering change.

Sources covered

What this session teaches, and where it came from

✓Breiman (2001), Statistical Modeling: The Two Cultures, sections 1 to 5, 9 and 11Statistical Science 16(3), 199-231; read at source, every quotation on the page checked against the published text
✓The ozone, chlorine, court delay and hepatitis examplesAll figures (450 variables, 30,000 compounds, 95 percent, 20.6 / 17.4 / 12.3 percent) quoted from the paper
◐The 98 and 2 percentBreiman's own estimate, labelled so in the paper; Efron disputes it in the same issue
✓The build-along on UCI Breast Cancer Wisconsin (Diagnostic)Wolberg, Mangasarian, Street and Street (1993), UCI Machine Learning Repository, doi 10.24432/C5DW2B, CC BY 4.0; every printed output produced by the code on scikit-learn 1.9.0
◐The Rashomon effect and the discussants' repliesNamed here, taught elsewhere: multiplicity in ML Epistemology, the four comments in session 6

Every fact on this page, and its verification tier, is recorded in the course's source map.

Session 1 cheat sheet · pin this

The one question"Are you modeling the inside of the box or not?" Breiman's own test for which culture you are in.
Data modellingAssume an equation generated y from x. Validate with fit tests and residuals: a yes or no.
AlgorithmicLeave the box shut. Find f(x) that predicts y. Validate with accuracy on unseen rows: a number.
The conclusions are about the modelNot about nature. A poor emulation gives conclusions that may be wrong.
Beat the base rate firstHepatitis: everyone survives 20.6 percent error, logistic 17.4. Biopsies: benign for all is 62.6 percent right.
Two numbers per modelHeld-out accuracy, and what it lets you say: 31 coefficients, or 1,673 unread questions.