The two-cultures bench

Held-out accuracy. Right answers out of the 171 biopsies neither model saw while fitting. Each biopsy is 0.58 points, so read the count, not the decimal.

Numbers to read the whole model (ochre). One coefficient per feature plus an intercept. The bench lists the three largest, standardised so they are comparable, with their signs: a minus leans towards malignant, because benign is coded 1.

Yes or no questions inside it (violet). Every internal node in every tree is one question of the form "is worst area at or below this threshold?". The deepest path is how many questions one biopsy may have to answer. The bench also shuffles one feature at a time across the scored rows and counts the biopsies lost: Breiman's own importance idea from his hepatitis example, done on the held-out rows. Importance says which features matter, never in which direction.

Sign surprises (ochre, small print). How many coefficients point against their own feature's correlation with benign on the training rows. A readable model is only as honest as what it lets you read; session 2 explains where these come from.

Disagreements. The biopsies on which the two cultures give different answers, and which culture was right on each. Every other biopsy gets the same answer from both, so on this data the entire argument lives in these few rows.

The regression standardises each feature on the 398 fitting rows, then minimises the log loss plus half the squared size of the weights (scikit-learn's default C=1 objective, intercept not penalised) by Newton's method until the step is below 10^-10. scikit-learn 1.9.0 gives the same held-out counts, with every coefficient within 0.000001 of the bench's.

The forest grows each tree on a bootstrap of the fitting rows drawn from a seeded generator, tries the square root of the number of features at every split (5 of 30, or 1 of 3), picks the split with the best Gini gain, and grows until every leaf is pure. The forest averages its trees' leaf shares and says benign above one half. A plain-Python copy of the same seeded algorithm reproduces every count on this page exactly.

The cross-engine tolerance. scikit-learn's own forest uses a different random stream, so it cannot match to the biopsy. Over its seeds 0 to 19 it scores 165 to 168 of 171 with all 30 features (median 167) and 165 to 169 with three (median 167). The bench's forest sits inside both ranges. Any forest-versus-regression gap smaller than that range is a gap between random draws, not between cultures.

The creators' documentation says the two classes are linearly separable using all 30 features. A boundary that is close to a flat plane is exactly what a logistic regression draws, so on this file the data modelling culture's guess about the inside of the box happens to be nearly right. Breiman's argument was never that the equation is always wrong; it was that nobody checks. Here the check, 170 of 171 on unseen biopsies, is the credential.

Notice also what three features do to each culture. The regression becomes a model a clinician could hold in one hand: four numbers, with worst area carrying most of the weight (standardised coefficient -4.63), and no sign surprises. The forest, given fewer features to choose between, grows larger, 2,700 questions against 1,647, because each tree has to carve the same boundary out of fewer directions.

Set the forest to one tree. It asks 19 questions, never more than 8 on one biopsy, and you could draw it on a whiteboard, which is Breiman's "A+ on interpretability". It is also 12 biopsies behind the regression, every one of them a biopsy the regression gets right: "a B on prediction". Ten trees close almost all of that gap with 152 questions, and a hundred add nothing further. The step from a readable tree to an unreadable forest is where the Occam dilemma of session 1 lives, and on this data it is paid for in the first ten trees.

With the labels shuffled, there is nothing true left to learn, so a model's best move is to call most biopsies benign. The regression lands on 116 of 171 and the forest on 114, against the do-nothing 107. The residue is what one particular shuffle leaves behind by chance: the bench uses one fixed permutation so every visitor sees the same numbers, and a different permutation would give a slightly different residue. The honest reading is the size of the fall, 54 and 53 biopsies, not the exact landing point. The forest, meanwhile, grows 6,730 questions deep into nothing, four times its honest size: a complex model will always find something to fit.

Nobody sets a switch that says "grade on the training rows". The peek arrives by degrees: tuning a setting until the test score improves, choosing among ten candidate models by their test score, re-splitting after a disappointing result. Hoadley names it in his comment on Breiman, "in industry and academia, there is sometimes a little tinkering, which involves peeking at the test sample. The result is some bias in the test sample or cross-validation results", and asks the question that has no easy answer: "How do you optimize your results and get an unbiased estimate of the generalization error?" Session 3 is about the discipline that answers it; session 5 is about the institution, the common task framework, that enforces it.