Where we start
Sessions 1 and 2 kept one habit without naming it: 171 biopsies were set aside before any model existed, and every score was read off them. Tonight names the habit and its author. Tukey did not invent the held-out set, but he drew the line it depends on: one kind of work where you are allowed to look at anything and change your mind as often as you like, and a second kind where you state one claim in advance and test it once. The two colours still mean what they meant: ochre for the data modelling culture, violet for the algorithmic culture. Tonight's figures are mostly about rows, not cultures, so terracotta does most of the talking.
★ What you walk out with tonight. Tukey's three-part test of a science, in his words, and why mathematics fails it; the difference between what the 1962 paper actually says and what later books and slides attribute to it; a clear rule for what exploration may do and what confirmation may do; and a pandas script that finds a rule on 398 training rows, writes it down, and grades it once on 171 untouched rows, next to two peeks that score higher and prove nothing.
A statistician who stopped being sure
6 min live"The Future of Data Analysis" is not a methods paper. It is a job description for a field that did not yet have a name, and it argues that the field's standard of truth should be experience, not proof. The whole claim rests on a three-part definition, and the figure below draws it as three gates.
LiveThe confession, and the three gates4 min▶
The paper's first page of argument is personal. "For a long time I have thought I was a statistician, interested in inferences from the particular to the general. But as I have watched mathematical statistics evolve, I have had cause to wonder and to doubt." (Tukey, 1962, p.2). He then says what he cares about instead: "my central interest is in data analysis, which I take to include, among other things: procedures for analyzing data, techniques for interpreting the results of such procedures, ways of planning the gathering of data to make its analysis easier, more precise or more accurate, and all the machinery and results of (mathematical) statistics which apply to analyzing data."
To claim data analysis is a science he needs a definition of science, and he gives one with three constituents (pp.5-6): "(a1) intellectual content, (a2) organization into an understandable form, (a3) reliance upon the test of experience as the ultimate standard of validity." Then the sentence that must have stung his readers: "By these tests, mathematics is not a science". Mathematics has content and organisation, but its standard of validity is proof. Data analysis, he argues, passes all three, and is "a science, one defined by a ubiquitous problem rather than by a concrete subject."
Gate a3 has a price, and he states it on p.6. "(b1) Data analysis must seek for scope and usefulness rather than security." (b2) It "must be willing to err moderately often in order that inadequate evidence shall more often suggest the right answer." (b3) It "must use mathematical argument and mathematical results as bases for judgment rather than as bases for proof or stamps of validity." Hold on to (b2): a discipline that is allowed to err moderately often needs somewhere to find out when it has.
Self-studyThe right question, sanctification, and a quote that is not there4 min▶
The most quoted sentence. Section 11, "Facing uncertainty" (pp.13-14): "Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise." D. R. Cox, replying to Breiman in 2001, gives a close cousin: "Better a rough answer to the right question than an exact answer to the wrong question, an aphorism, due perhaps to Lord Kelvin". Same idea, two attributions; Cox's "perhaps" is his, and nobody has settled the origin here. How you choose the right question in the first place is its own discipline, taught in Data Thinking, session 3 ↗.
The danger he names. On p.13 Tukey warns against "the use of formal data-analytical procedures for sanctification, for the preservation of conclusions from all criticism, for the granting of an imprimatur." A significance test run to bless a conclusion already reached is exactly that. Keep the word; Part 3 shows what sanctification looks like in numbers.
The attribution trap. David Donoho's 2017 "50 Years of Data Science" quotes a list, "Four major influences act on data analysis today", right after his quotations from this paper; a full-text search of the whole 1962 paper, pages 1 to 67, finds no such list, so it can only be cited as a list Donoho attributes to Tukey, source unconfirmed. That is the professional habit this session is about, applied to text: a quote is a hypothesis until you have read it at source.
"Live with the data before you plunge into modeling." Leo Breiman spent thirteen years as a free-lance consultant before joining Berkeley in 1980, on projects such as forecasting ozone for the EPA from more than 450 weather variables. When he left consulting, this was the second of the five perceptions he took back to the university (Breiman, 2001, p.201). Tukey's closing line in 1962 asks for the same temperament from a whole profession: "the rocky road of real problems in preference to the smooth road of unreal assumptions, arbitrary criteria, and abstract results without real attachments. Who is for the challenge?" (p.64).
The detective and the judge
5 min liveThe popular story is that Tukey's 1962 paper launched exploratory data analysis. Read at source, it does something subtler. It never uses the words "exploratory data analysis" as a named practice, and it keeps the other half on purpose: Tukey says we should go on "using statistical techniques for the confirmatory appraisal of observations through such conclusion procedures as confidence statements and tests of significance" (p.7). The named pairing came later, and the dates matter when you quote him.
LiveThree dates, two tiers of evidence2 min▶
1962, read at source. Data analysis as a science; "confirmatory appraisal" kept, once, on p.7; the words "exploratory data analysis" do not appear as a named practice. On the same page Tukey quotes Martin Wilk: "The hallmark of good science is that it uses models and 'theory' but never believes them."
1977, reported. The book Exploratory Data Analysis (Addison-Wesley) is widely described as casting exploration as detective work. It was not opened for this session, so treat that description as reported.
1980, reported, title only. "We Need Both Exploratory and Confirmatory", The American Statistician 34(1). The title itself is the claim tonight teaches; the text was not opened here.
Donoho, writing in 2017 and read at source, is where the detective metaphor reaches today's data science: "Since John Tukey's coining of the term 'exploratory data analysis' (EDA), we all agree that every data scientist devotes serious time and effort to exploring data ... Such detective work adds crucial insights to every data-driven endeavor."
LiveWhat exploration is allowed to do1 min▶
Anything, as long as it happens on the exploration rows. Plot every feature. Compare class means. Try every cut on every column. Change your mind ten times. Tukey's own words from p.4 describe the attitude: "we need to stress flexibility of attack, willingness to iterate, and willingness to study things as they are, rather than as they, hopefully, should be."
What exploration produces is not a result. It is a hypothesis, and its training score is a rumour: the rows that suggested the rule are the rows that score it, so the score is flattered by construction.
Self-studyWhat confirmation is allowed to do2 min▶
One pre-stated test, once. Before the confirmation rows are opened, write down the rule, the metric and what would count as failure. Then run it and report what it printed, even when it is disappointing.
Everything else is off limits: re-choosing the cut, swapping to a feature that "looks better here", dropping the rows that disagree, or rerunning with a new seed until the number is pleasant. Each of those moves a decision to the wrong side of the valve, and the rows stop being a judge. If you genuinely learned something from the confirmation, the honest move is to treat it as exploration and find fresh rows to confirm on. What a single test score then entitles you to claim is a question with its own session: ML Epistemology, leader session 1 ↗.
Two years nobody touched. In Breiman's EPA ozone project in the mid-1970s, the job was to predict the next day's ozone in the Los Angeles Basin from over 450 meteorological variables. Seven years of data were available. The first five years were used to build the predictor; the last two were set aside as the test set (Breiman, 2001, pp.200-201). Whatever happened on the five years was detective work. The two years were the judge.
A peek is a search the judge helped with
4 min liveNobody sets out to cheat. The damage comes from small, sensible-looking moves: glance at the test score, adjust one setting, glance again. Bruce Hoadley, of the credit-scoring company Fair, Isaac, described it in his 2001 reply to Breiman as "The algorithm tuning dilemma", and it is the reason the rows that confirm must be sealed before the exploring starts.
LiveThe tuning dilemma, and the drift behind it2 min▶
Hoadley (2001, p.223): "in industry and academia, there is sometimes a little tinkering, which involves peeking at the test sample. The result is some bias in the test sample or cross-validation results." And the question every modelling team faces: "How do you optimize your results and get an unbiased estimate of the generalization error?" The answer is not to stop tuning. It is to tune on rows that will never grade you.
He adds a second reason the judge must be honest: even a clean test score "does not guarantee high performance on future samples, things do change" (pp.221-222), and Fair, Isaac redevelops its core models about every 18 to 24 months. An inflated number today is also the baseline you will measure next year's drift against.
The field later built this seal into its institutions. In Donoho's account of the common task framework, the referee's test set "is sequestered" and scored automatically, so no competitor can peek. Session 5 reads that history in full.
Self-studyWhat the peek does to our two models2 min▶
The session 4 bench has a "peek" rung: grade each model on the 398 rows it was fitted on instead of the 171 it never saw. The forest scores 398 of 398 (100.0 percent) on the rows it memorised, against 167 of 171 (97.7 percent) held out. That gap is what peeking hides.
The regression goes the other way: 392 of 398 on training (98.5 percent) but 170 of 171 held out (99.4 percent). A held-out set of 171 rows can flatter as well as punish: one biopsy is 0.6 points. The lesson is not that held-out scores are exact. It is that they are the only scores nobody has had a chance to steer, so they are the only ones allowed to carry the claim, with their small-sample wobble stated beside them.
A later judge, three times over. Hoadley's Fat Scorecard, the credit model whose 162 score weights had confidence intervals he called "useless", was not left to its original test sample. It had three extra test samples taken one, two and three years later (Hoadley, 2001, p.222), and it held up. That is what an honest later confirmation looks like: fresh rows, gathered after the model was fixed, that nobody could have tuned against.
Explore the 398, confirm on the 171 once
★ 22 min · everyone buildsSame biopsies, same split as sessions 1 and 2: the UCI Breast Cancer Wisconsin (Diagnostic) set, 569 biopsies, 30 nucleus measurements each, split once with seed 2001. This time there is no model library. You explore the training rows with pandas, hunt for the best single-feature rule by hand, write the hypothesis down, open the held-out rows once, and then commit the two classic sins on purpose so you can see what they print. Every output below is what the code prints.
Load the biopsies and make the same split as session 1. Thirty percent of each class is set aside with seed 2001. From here until step 5, the variable test is a list of row numbers you do not look at.
Explore: class means, training rows only. Put the 398 rows in a DataFrame and compare the two classes on three features. Malignant nuclei are bigger and more concave; that is the first clue, not a conclusion.
Explore harder: the best single-feature rule. For every feature, try every observed value as a cut, in both directions, and count how many training biopsies the rule gets right. Keep the top three. This is a brute-force search, and on the exploration rows it is completely allowed.
Write the hypothesis down before the 171 exist for you. This step prints nothing and matters most. Worst radius and worst area tie on 367; the sort keeps worst radius first, and you commit to it now. State the rule, the metric and the bar it has to clear, in a comment that will still be in the file when someone reviews it.
Confirm once. Apply the written rule, unchanged, to the 171 held-out biopsies. It is lower than the 367 of 398 it earned in exploration, and that drop is information: it is what a training score was hiding.
Sin on purpose, small: re-choose the cut on the held-out rows. Same feature, but let the 171 pick the threshold. It gains one biopsy. That one biopsy is not evidence about worst radius; it is the held-out rows fitting themselves.
Sin on purpose, big: search every feature and cut on the held-out rows. Now the 171 choose the feature as well. The winner scores 163, the best number you will see tonight. Then check it on the 398 rows it never chose from: it does worse there than the worst radius rule you found honestly.
Report 157, not 163. The peeked rule looks six biopsies better on the 171 and is thirteen biopsies worse on the 398 than the rule you wrote down. The held-out rows stopped being a judge the moment they chose the feature. If you want to know whether worst concave points really is better, that is now a hypothesis, and it needs rows that have not voted yet.
Before session 4
30 min- Read the first seven pages of Tukey's 1962 paper, through the "confirmatory appraisal" passage on p.7. Mark every sentence about what data analysis should be allowed to do, and every sentence about what it must still check.
- Check one quote at source. Take a Tukey, Box or Breiman quotation you have seen on a slide or a post. Find it in the primary text. If you cannot, rewrite your citation as "attributed to" or "reported", the way the four major influences list has to be cited.
- Run the valve with a different hypothesis. Before touching the 171, write down worst area <= 867.1 as your rule in the step 4 comment, then run step 5 for it once. Write down what it prints, and do not change the rule afterwards.
- Find a peek at work. Pick one number from a recent report or deck and ask who chose the model, the threshold or the segment, and on which rows. If the same rows chose it and scored it, write the one-line hypothesis that should have been written first.
Three questions
Clear all three and session 3 gets stamped in your passport.
1 · A slide says "Tukey coined exploratory data analysis in his 1962 paper and listed four major influences on data analysis." What does the 1962 paper, read at source, actually contain?
The 1962 paper is the programme; the named practice came later. A secondary source quoting a list right after a paper's quotations is not proof the list is in that paper, which is why a quote stays a hypothesis until it is read at source.
2 · In the build-along, a search over all 30 features on the held-out rows finds worst concave points at 163 of 171, beating the written-down rule's 157. Why is 163 not a result?
A score is only a judge's verdict when the judge had no hand in picking the defendant. Hoadley calls the mild version "a little tinkering, which involves peeking at the test sample", and says the result is bias.
3 · The logistic regression scores 392 of 398 on its own training rows and 170 of 171 held out. Why can the held-out score beat the training score?
Held-out scores are unsteered, not exact. Redrawing the split until it looks right would be a peek of its own, the same sin as step 7 with a different lever.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 3 cheat sheet · pin this
Tonight you graded one hand-found rule. Session 4 puts both cultures on the bench: the same 171 sealed biopsies, six rungs you can switch between, from all 30 features to the creators' three, a single tree, shuffled labels and the peek you met tonight, with every number read off rows that neither model has seen.