Where we start
Four sessions in, you have Breiman's two cultures, Box's useful wrong models, Tukey's explore-then-confirm discipline, and a bench that put both cultures on the same rows. Donoho's paper is the one that ties them into a history. It is also the one that names the institution most practitioners live inside without noticing: the shared dataset, the held-back test set, and the leaderboard. As always, ochre is the data modelling culture and violet is the algorithmic culture; Donoho's own names for them are "generative" and "predictive".
★ What you walk out with tonight. The fifty-year arc from Tukey to Donoho with the right dates on it; the three ingredients of a common task framework, precisely enough to spot one at work; the six divisions of what Donoho calls greater data science, and where both of Breiman's cultures sit inside them; and a working referee with a sealed test set, a sorted leaderboard, and a measured demonstration of why the best of 20 submissions is not an honest score.
From Tukey's paper to Donoho's
6 min liveDonoho's paper was prepared for the Tukey centennial and opens from Tukey's 1962 "The Future of Data Analysis", the paper session 3 read at source. What follows is a chain of statisticians telling their own profession to widen, each in print, each largely ignored. Donoho's twist is his explanation of why the widening finally happened.
LiveThree exhortations, and why they did not work4 min▶
Tukey, 1962. The starting line is a confession, read at source in session 3: "For a long time I have thought I was a statistician, interested in inferences from the particular to the general. But as I have watched mathematical statistics evolve, I have had cause to wonder and to doubt." Data analysis, he argued, was "a science, one defined by a ubiquitous problem rather than by a concrete subject."
Chambers, 1993. Chambers' "Greater or Lesser Statistics, A Choice for Future Research" put the choice in its title: widen to everything involved in learning from data, or stay narrow. Quoted here via Donoho.
Cleveland, 2001. William Cleveland's "Data Science: An Action Plan for Expanding the Technical Areas of the Field of Statistics" went further and wrote a budget. Six areas, with the share of a department's effort each should get, as Donoho quotes them:
- Multidisciplinary investigations, 25 percent
- Models and methods for data, 20 percent
- Computing with data, 15 percent
- Pedagogy, 15 percent
- Tool evaluation, 5 percent
- Theory, 20 percent
Breiman, 2001. The paper sessions 1 to 4 read. Donoho renames the cultures "generative" and "predictive" and says so in footnotes, so when you read him, generative means Breiman's data modelling culture and predictive means the algorithmic one. His reading of the discussion: Cox and Efron objected, Hoadley engaged enthusiastically.
Donoho's point is that the exhortations had little effect on their own. What made a wider field concrete was code: the S language, then R, then shared workflows that let people actually compute with data rather than be told they should. Keep that in mind for Part 2, because the common task framework is the same kind of explanation: an institution, not an idea, did the work.
Self-studyA list Donoho attributes to Tukey, and a lesson in checking quotes3 min▶
Right after his quotations from Tukey's 1962 paper, Donoho quotes a list that begins "Four major influences act on data analysis today": formal theories, computers and display devices, larger bodies of data, and quantification. It reads like Tukey and sits among genuine Tukey quotations. A full-text search of the whole 1962 paper, pages 1 to 67, read at source for these sessions, does not find it.
That does not mean Donoho invented it. Tukey wrote a great deal, and the list may come from a later paper. It means the attribution is unconfirmed, so it can be quoted only as "a list Donoho attributes to Tukey". The same paper says "Since John Tukey's coining of the term 'exploratory data analysis' (EDA)"; also true in spirit, but the words "exploratory data analysis" as a named practice are not in the 1962 paper either. They arrive with the 1977 book (reported).
The professional habit this teaches: a secondary source quoting a primary one is a pointer, not a check. When a quotation matters to your argument, open the primary text, and when you cannot, say which source you are trusting.
A department budget nobody adopted. Cleveland's 2001 plan is unusual among manifestos because it is a spending plan: a quarter of the effort on multidisciplinary investigations, 15 percent on computing with data, and only 20 percent on theory. Hold it next to what a statistics department of the time taught and you can see why Donoho calls these papers exhortations. They told the field where to go. The fields that went there, he argues, got there by writing software and running contests.
The secret sauce is a referee
5 min liveThe term is Mark Liberman's; Donoho gives it the strongest line in his paper: "It is no exaggeration to say that the combination of a predictive modeling culture together with CTF is the 'secret sauce' of machine learning." And he says the framework "is the single idea from machine learning and data science that is most lacking attention in today's statistical training."
LiveThe three ingredients, and Liberman's three lessons3 min▶
As Donoho sets it out, a common task framework has exactly three parts:
- (a) A publicly available training dataset, with a list of features and a class label for each observation.
- (b) A set of enrolled competitors whose common task is to infer a prediction rule from the training data.
- (c) A scoring referee, to which competitors submit their rules. The referee runs each rule against a test set that is sequestered behind a wall, and reports the score automatically.
Take away any one and it stops working. Without (a) nobody can enter; without (b) there is no pressure; without (c) every team grades itself, and session 3 showed what a small peek at the confirmatory rows does to a score.
Liberman's experience, as Donoho reports it, came down to three lessons. Error rates decline by a fixed percentage each year, towards an asymptote. Progress comes from many small improvements, not a breakthrough: "a change of 1% can be a reason to break out the champagne". And shared data gets reused in ways nobody planned.
What a benchmark score entitles you to claim is its own question, and it belongs to ML Epistemology, leader session 4 ↗. Tonight is only the mechanism.
Self-studyHow the framework was born, as Liberman tells it2 min▶
This history is reported by Donoho from Liberman, not read at any primary source here, so treat each step as their account. In the 1960s a review led by J. R. Pierce sent US machine translation research "essentially to zero for decades". The way back ran through Jelinek's team at IBM, working from the Canadian Hansards. In the late 1980s DARPA adopted the common task framework for its language research, with NIST as the referee.
Notice what the story is about. It is not a better algorithm. It is a funding agency deciding that claims would be settled by a shared dataset and a neutral scorer, rather than by whoever wrote the most persuasive paper.
The Netflix Challenge. A $1M prize run on the three ingredients: shared training data, enrolled teams, and a held-back set scored by the sponsor. Donoho uses it as the framework's best-known public case, and adds a footnote that is the most useful sentence in the section for anyone who sponsors a contest: the winning algorithm was never implemented. A leaderboard can tell you which rule scores best on the sealed rows. It cannot tell you whether that rule is worth running.
A map of the whole field
4 min liveDonoho's answer to "what is data science" is a map with six divisions, which he calls greater data science. Its most useful property for tonight is where it puts the argument of the last four sessions: both of Breiman's cultures fit inside one division, and five others surround them.
LiveSix divisions, one split box, and a hedge3 min▶
The six divisions, by name: Data Gathering, Preparation, and Exploration; Data Representation and Transformation; Computing with Data; Data Modeling; Data Visualization and Presentation; Science about Data Science. Cite them by name, because the published paper numbers them two ways: the list puts Modeling fourth and Visualization fifth, while the section headings that follow read "GDS4: Data Visualization and Presentation" and "GDS5: Data Modeling". Anyone who writes "GDS4" is telling you which of the two they read.
Data Modeling holds both cultures. Donoho's sentence: "Each data scientist in practice uses tools and viewpoints from both of Leo Breiman's modeling cultures". In his map the two-cultures argument is a disagreement inside one division, not a split of the field.
The 80 percent is a hedge. Under the first division Donoho writes "Some say that 80% of the effort ..." goes into gathering, preparing and exploring data. That is "some say", not a measurement, and it should never be repeated as a statistic. He also claims the first division "is more important than" modeling "as measured using time spent by practitioners", while modeling dominates academic teaching. That is his claim; nothing here measures it.
His one-line definition, from the conclusion: "data science is the science of learning from data".
Self-studyData science in 20652 min▶
Donoho ends with a forecast, and one sentence of it points straight back at Part 2: "Many new common task frameworks will appear; however, the new ones would not always have prediction accuracy for their performance metric."
Read it with Breiman in mind. The referee's score is where the predictive culture gets its authority, and Donoho expects the score itself to change: a framework could grade something other than accuracy on the sealed rows. What else could a referee grade? Session 6 gives four eminent statisticians' answers from 2001, and they do not agree.
Put your own week on the map. Take last week's work and tag each hour with one of the six divisions. If most of it lands in the first one, you are in good company on Donoho's reading, but you have also produced the only measurement on this page of his "time spent by practitioners" claim, and it is a sample of one.
Run a common task framework on 569 biopsies
★ 22 min · everyone buildsSame biopsies, same split as session 1: 398 training rows and 171 held-out rows, split once with seed 2001. Tonight the held-out rows go behind a wall. You write the referee, submit four rules from earlier sessions, print the leaderboard, and then do what every competitor eventually does: submit the same kind of model 20 times and keep the best. Every output below is what the code prints on scikit-learn 1.9.0.
Split once, then seal. The split is session 1's, byte for byte. The only new line turns the held-out indices into a set that only the referee will read. Competitors get X[train] and y[train] and nothing else.
Write the referee. It takes a prediction function, not predictions: a competitor hands over a rule and never touches the sealed rows. It returns one number, the count of right answers, which is ingredient (c) in six lines.
Submit the do-nothing rule first. Every leaderboard needs its floor. Calling every biopsy benign is a rule the referee can score like any other.
Submit session 3's rule. One feature, one cut, chosen by exploring the 398 training rows: benign if worst radius is at most 16.76. Session 3 confirmed it once on the held-out rows; here the referee does the confirming.
Submit both cultures and print the board. The same two models as session 1. The logistic regression is wrapped in a pipeline so the scaler is fitted inside the rule, on training rows only. How the forest works is taught in Ensemble Methods, session 2 ↗.
Now climb the leaderboard. Submit the forest 20 times, changing nothing but its seed, and keep the best. No team would call this cheating; it is what tuning against a public score looks like, and it is session 3's peek done politely, one submission at a time.
Report the median, not the maximum. Nothing about the model changed across those 20 submissions, yet the kept best is 168 against a median of 167.0. That one biopsy is optimism bought by submitting 20 times, and on a public leaderboard, where every team does the same, it compounds. The sealed rows stay sealed only while each team's number of looks stays small.
Before session 6
30 min- Read Donoho's sections on the common task framework and greater data science in JCGS 26(4). Find the footnotes where he renames Breiman's cultures, and write down the exact words he uses for each.
- Add a submission limit to the referee. Make it refuse after three calls per team name, then repeat step 6 with a limit of three. Write down what the best kept score can and cannot be. Your numbers will differ depending on which three seeds you pick.
- Find a common task framework at work or in your field. Name its (a), (b) and (c). If one is missing, say which, and what fills the gap.
- Tag one week of your hours by the six divisions, by name. Bring the share that landed in Data Gathering, Preparation, and Exploration, and be ready to say why a sample of one does not test Donoho's claim.
Three questions
Clear all three and session 5 gets stamped in your passport.
1 · Which list is the common task framework, as Donoho sets it out from Liberman?
The three ingredients are institutional, not algorithmic. Remove the sequestered test set and every team grades itself; remove the public training data and nobody can enter. Donoho calls the pairing of this framework with the predictive culture the "secret sauce" of machine learning.
2 · A team submits the same forest with 20 different seeds and reports its best score, 168 of 171. Why is that number optimistic?
Choosing the best of many looks at the sealed rows turns the referee into part of the training loop. The honest report is the median, or better, a fresh sealed set the team has never scored against.
3 · In Donoho's six divisions of greater data science, where do both of Breiman's cultures sit?
The two-cultures argument is a disagreement inside one division, not a split of the field. And the 80 percent is Donoho's "some say", a hedge, never a measurement.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 5 cheat sheet · pin this
Five sessions have handed you two cultures, three checks for a useful model, a sealed test set and a leaderboard. Session 6 asks the practical question: faced with a real problem, which culture is this question? It lets the four 2001 discussants, Cox, Efron, Hoadley and Parzen, argue it out in their own words, and it does not pick a winner.