Where we start
Five sessions of doing the work carefully. Tonight is the other ninety percent of a working life, which is receiving work somebody else did and deciding whether to believe it. The good news is that you already know what to ask, because every question below is one of the earlier sessions turned around and pointed at somebody else's output.
★ What you walk out with tonight. Six questions you can ask out loud without being the difficult person in the room, one real analysis read properly, and a final scorecard on everything the course has claimed.
Where to go next. If the answer to "is this difference real or noise" starts mattering, that is learn-statistics-with-phoebe. If you need to design the comparison rather than inherit it, learn-experimentation-with-phoebe. If the chart itself is the problem, learn-dataviz-with-phoebe. And when a question genuinely needs a model, learn-intro-ml-with-phoebe is the next rung.
Six questions, in the order that finds things fastest
7 min liveOrdered by how much they typically find per second spent. The first three are almost always worth asking; the last three are for when the decision is expensive.
LiveThe first three, and what each one catches4 min▶
Compared with what? A conversion rate of 4 percent is neither good nor bad until you know what it is being held against: last quarter, the other channel, the target, the industry. Numbers presented without a comparison are asking you to supply one from memory, and you will supply a flattering one.
Who is missing from this? Charts are made from rows that survived. Customers who churned before the survey, orders that errored out, the branch whose data feed broke in March. The rows that dropped out never appear, and their absence looks exactly like them not existing. This is the question that catches the analysis where a whole group silently left.
Did the mix change? Four words, ten seconds, and the single highest-yield question in the set. If the composition of the population shifted between the two periods being compared, the headline is a blend of two things moving at once, and it can point the opposite way to every one of its parts. Session 4 showed exactly that: a headline up thirteen pence while email orders fell by GBP 3.83 and social orders by GBP 2.97.
Self-studyHow to ask without becoming the difficult person3 min▶
All six questions can be asked in a way that sounds like interest or like an accusation, and the difference is almost entirely whether you attach a reason. "Did the mix change?" alone sounds like a challenge. "Did the mix change - I ask because we shifted spend to social in that window and it would pull the average up on its own" sounds like somebody helping.
Two more habits that keep it collegial. Ask about the work rather than the person: "what does this compare against" rather than "why didn't you compare it". And ask your own numbers the same questions in public, visibly, so that the six questions read as house style rather than as something that happens to other people's work.
The one exception is when the decision is large and irreversible. Then the questions get asked properly, in writing, and the politeness budget goes on tone rather than on skipping any of them.
The story around a number travels further than the number
4 min liveTwo examples from inside these six sessions, because the failure being described is one these pages could have committed and nearly did.
| The version that circulates | What is actually the case | What went wrong |
|---|---|---|
| Data scientists spend 80% of their time cleaning data | One 2016 survey reported cleaning at 60% and collecting at 19%; the 80% appears only by adding two different answers. Other years and an independent survey give quite different, lower figures, and none measured time use | Four citations deep, nobody re-measured, two tasks merged into one number |
| Berkeley was sued for sex bias in admissions, and the lawsuit collapsed when the data was broken down | The admissions figures and the reversal are real. The lawsuit appears not to be: the graduate division feared being sued and commissioned the analysis. No suit was filed | A memorable story attached itself to a real result and has been repeated in at least six textbooks and papers |
LiveWhy the second one is the more useful lesson3 min▶
The Berkeley numbers are correct and the conclusion drawn from them is correct. What is wrong is the narrative wrapped around them, and the narrative is the part everybody remembers and repeats. That is worth sitting with: the single most-cited example of a misleading aggregate is itself surrounded by a claim that nobody checked, in textbooks, for decades.
It is also worth noticing what the careful version of the finding actually was. The authors did not conclude "no bias". They found four departments significantly biased against women and six significantly biased against men, and their pooled figure showed a small but statistically significant bias in favour of women. Nearly every retelling flattens that into something cleaner than what the paper said, in the same way that the 80 percent claim flattens two survey answers into one number.
The pattern is the same in both cases and it is the thing to watch for: the tidy version travels and the careful version does not. When you hear a number with a good story attached, the story is the part most likely to be wrong.
Self-studyThe tiers these sessions used on themselves3 min▶
Every factual claim across these six sessions carries a tier. Read directly from the primary source, read from a reliable secondary that quotes the primary, or seen only in a search result. The Berkeley figures above are the middle tier: the 1975 paper itself would not load on any attempt during this build, so the numbers come from three independent sources that reproduce its tables and agree with each other to the digit. That is good enough to teach from and not good enough to call first-hand, and the difference is stated rather than smoothed over.
You can do the same thing in your own work at almost no cost. When you quote a figure, write next to it where you got it and whether you saw the original. Three months later, when somebody challenges it, that one line is the difference between defending your number and discovering you cannot.
Read one, then score the course
★ 22 min · a real analysisThe chart or slide you brought. Twelve minutes reading it properly, then ten on the scorecard.
State the claim in one sentence, in your own words, as an outcome and a direction. If you cannot, that is already the finding: the analysis has not told you what it thinks is true.
Ask the first three of the chart rather than the person: what is this compared with, who could have dropped out before this was drawn, and did the composition change between the periods shown. Write what you can answer from the page alone.
Ask the last three. What is the denominator and is it the right one; how many links back is the source; and what result would have changed the author's conclusion.
Write the one question you will actually ask, with its reason attached. One is the right number: it gets answered, whereas six at once gets defended.
Score the card below honestly, including the rows you know you skipped.
If you take one thing from the whole course. Most wrong answers are answers to the wrong question, and most of the rest are answers nobody checked twice. Both failures are cheap to prevent and expensive to discover later, and neither needs any mathematics.
After the course
25 min- Ask your one question and note what comes back. The answer is often "I do not know", which is not a failure of the analysis; it is the beginning of the next one.
- Put the six questions somewhere you will see them before your next review meeting. They take ninety seconds and they are the whole of the six sessions in a form you can use while somebody is talking.
- Trace one more number to its source. Every organisation has three or four figures everybody quotes and nobody can place. Finding the origin of one is a genuinely useful afternoon.
- Pick the next course by what is actually blocking you: whether a difference is real, how to design the comparison, whether the chart is honest, or whether the question finally needs a model.
Three questions
Clear all three and session 6 gets stamped in your passport.
1 · You have ninety seconds with a chart in a meeting. Which three questions do you ask first?
The first three are answerable from the page itself and catch the largest problems: no comparison, a group that silently dropped out, and a composition shift that makes the headline a blend of two moving things. Significance and intervals matter, and they are a later rung and a different course.
2 · The Berkeley admissions case: the figures and the reversal are real, but the widely told lawsuit story appears to be false. What is the lesson?
The numbers survive scrutiny; the narrative did not. The same flattening happened to the paper's actual conclusion, which was more careful than the retellings, and to the 80 percent claim, where two survey answers became one number. The pattern repeats because tidy stories are easier to carry.
3 · You have six good questions about a colleague's analysis. What do you actually do?
Attaching the reason is what turns a challenge into help: "did the mix change, I ask because we moved spend to social that month" invites an answer. Volume triggers defence. The exception is genuinely expensive decisions, where the politeness budget goes on tone rather than on skipping questions.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 6 cheat sheet · pin this
You can take a vague request, work out what would answer it, look at the data honestly, check a number at more than one level, stop at the right rung and read somebody else's work. What blocks you next decides the door: whether a difference is real or noise is Statistics ↗, designing the comparison instead of inheriting it is Experimentation ↗, whether the chart tells the truth is Data Visualization ↗, and a question that genuinely needs one is Intro to ML ↗.