Where we start
Session 4 was about checking a number harder: four levels, one headline that held, one that reversed from plus thirteen pence to minus GBP 3.57 once the channel mix was held fixed, and one question with no answer in the rows at all. Tonight is the opposite pressure. Daybreak's promotion is still the running situation and the same three people are still asking, but the question now is how much machinery any of it actually deserved. Most questions worth asking are answered by a count. Reaching past one costs time, credibility and the ability of anybody in the room to check what you said.
★ What you walk out with tonight. One complicated number from your own work demoted to the simplest thing that would have answered the question it was asked, with both answers written side by side. Or, if it genuinely could not be demoted, a defensible written reason why, which is the same output and just as useful.
Where this sits on the shelf. When you genuinely do need a model, that is learn-intro-ml-with-phoebe. When you need an interval or a test, that is learn-statistics-with-phoebe. When you need a designed experiment, that is learn-experimentation-with-phoebe. All three are real work with real answers behind them. Here the subject is the step before any of them, which is deciding that you do not need one of them yet.
The ladder, and how far up it you actually need to go
7 min liveFive rungs, ordered by what they cost rather than by how impressive they sound. Every rung answers a different shape of question, and the rung above never answers the one below better. It answers a different one, more slowly.
LiveClimb only until the question is answered4 min▶
The rule is one sentence: stop at the first rung that answers the question that was actually asked. Everything difficult about applying it lives in that last clause, because you have to know what was asked before you can tell whether it has been answered, and that is session 1's work.
The four slots are what point at the rung. An outcome that is a yes or a no needs a count. An outcome expressed as a share needs a rate. A decision slot that says "run it again or not" needs a comparison, because a comparison is exactly the shape of that decision: this against that, one wins. None of those three needs a model, and no model will make any of them more decidable.
Where you genuinely climb is when the question changes shape. "Which of these eleven things is driving it" is a many-things-at-once question and it lives at rung 5. "How sure are we that the difference is real rather than noise" is a confidence question and it also lives at rung 5, in a different course. If your question is neither of those, the climb is not buying you anything, and it is worth saying that out loud before you start rather than after you finish.
Run it against Daybreak. The head of marketing asks whether conversion rose enough to repeat the offer. That is a rate compared across two periods: rung 3, one hour. Session 4 climbed to rung 4 on it and found the answer had not moved, which is a real result and cost twenty extra minutes. Nobody needed rung 5 for any of the three questions, and nobody ever proposed it, because the questions were written down first.
Self-studyWhy people climb too far4 min▶
Four honest reasons, none of them stupidity, and all four are worth recognising in yourself rather than in a colleague.
A model looks like more work and therefore more value. Three days of method produces an artefact somebody can see you built. A count produces one line. The effort is legible in one case and invisible in the other, and effort is what most organisations actually reward.
A count feels too simple to present. Standing up in front of eight people to say "it was 412, and that is 0.8 percent" feels like you have not done your job, even when it is the complete answer. This feeling is very strong and it is wrong, and naming it is most of the cure.
The tooling makes the elaborate thing easy to reach for. Fitting a model is now four lines, and the four lines are on the first page of every tutorial you have ever read. The path of least resistance genuinely runs uphill on this ladder, which is a recent and slightly strange state of affairs.
Nobody was ever criticised for being too rigorous. The failure mode is invisible, so it carries no social cost, so it never gets corrected.
Now the cost, plainly, because it is real and it is paid by somebody. You spend time you did not have. You produce a result that nobody in the room can check, which means it gets accepted rather than believed. And you produce a number whose failure modes are invisible to its audience: when a count is wrong, somebody notices, because they can recompute it over lunch. When a model is wrong, it is wrong quietly, and it stays in the deck.
| Rung | Answers | Roughly costs | Daybreak example |
|---|---|---|---|
| 1 A count | How many were there? | Minutes | How many orders came through the promotion at all |
| 2 A rate | How many out of how many? | Minutes | What share of those orders converted |
| 3 A comparison | This against that, or before against after | An hour | Conversion in the twelve weeks after against the twelve before |
| 4 A breakdown | The same comparison inside groups | Half a day | That same comparison inside email and social, which is where the revenue reversal showed up |
| 5 A model or a test | Many things at once, or how confident you can be | Days, and a different course | Not needed for any of the three questions anybody actually asked |
A count becomes useful the moment it has a denominator
5 min liveA bare count sits on a slide and nobody can do anything with it, because there is nothing to hold it against. The denominator is the part that turns it into something two people can argue about productively, and it is the part that gets left out.
LiveThe denominator is the part that gets left out4 min▶
A count alone is almost never an answer, and the reason is mechanical rather than stylistic. An answer has to be comparable with something, either with the same quantity at another time, or with a different group, or with a target somebody set. A bare count is comparable with nothing, because the thing it is counting out of has been left unstated and may well have moved.
The denominator is what restores that. Two periods become comparable, two channels become comparable, two companies of very different sizes become comparable, all because you divided by the number of chances the thing had to happen.
Here is the part that gets skipped: choosing the denominator is a judgement, and it belongs to the person who understands the business, not to whoever wrote the query. Complaints per order and complaints per customer are different measures of different things, and they can move in opposite directions in the same quarter. There is no technical fact that decides between them. There is only the question being asked, which is why session 1 comes before this one and not after it.
Self-studyThree denominators that change the story3 min▶
Per customer. Divides by the people involved. This is the right denominator when the question is about individual experience: is a typical customer having a worse time than they were?
Per order. Divides by the transactions. This is the right denominator when the question is about the process: is each trip through the system going worse than it was?
Per opportunity-to-happen. Divides by the number of chances the thing had to occur at all, which is often neither of the above. Delivery complaints belong over deliveries, not over orders, because orders that were never delivered could not generate one.
The same numerator over these three denominators can point three different ways in the same period, and none of the three is wrong. Which is exactly why stating which one you used is part of stating the answer, not a footnote to it. "Complaints fell" is not a finding. "Complaints per delivery fell from 1.1 percent to 0.8 percent" is one, and it survives somebody asking what you divided by.
Complaints rose forty percent quarter on quarter, which sounds alarming, and somebody will have put it on a slide in red. Then you notice orders rose sixty percent in the same period. The complaint rate fell. Both sentences describe the same two columns of the same table, and only one of them is an answer to "are we getting worse".
Two questions that tell you to stop
4 min liveKnowing the ladder exists does not tell you where to get off it. Two questions do, and both of them can be answered before any work starts rather than after it finishes.
LivePrecision that changes nothing is decoration3 min▶
If the decision is the same whether the answer is 12 percent or 18 percent, then spending three days narrowing a range down to a point is work that nobody needed and nobody will notice. The decision was already made by the time you had the range.
This sounds like a matter of taste and it is not, because the four-slot decision line from session 1 makes it checkable. You already wrote down what changes depending on the answer. So write the two candidate answers, the optimistic one and the pessimistic one, and ask what the decision slot says in each case. If it says the same thing twice, you are finished, and you were finished before you started.
The version of this that costs the most is the one where nobody ever wrote the decision down. Then there is no way to tell whether more precision would matter, so more precision always feels safer, and the work expands until somebody runs out of week. That is not rigour. It is an unanswered question about the fourth slot, showing up three weeks later as a scheduling problem.
Self-studyCheckability is a feature, not a limitation3 min▶
An answer the asker can verify gets trusted, acted on and remembered. An answer they cannot verify gets accepted politely and then quietly ignored, and you usually do not find out which happened.
A count and a rate are checkable by anybody holding the same spreadsheet. They can filter the rows themselves, get your number, and from that moment they believe it in a way that no amount of explanation produces. They can also find your mistake, which is uncomfortable for an afternoon and enormously valuable over a year.
That is a genuine advantage of the bottom of the ladder, not a consolation prize for having done something simple. It is the difference between a number the organisation now owns and a number that belongs only to you. And when the organisation owns a number, it starts appearing in other people's work, which is the actual goal.
None of this is an argument against rung 5. Some questions live up there and cannot be pulled down. It is an argument for knowing what you are giving up when you climb, so the climb is a decision rather than a reflex.
Demote one number
★ 22 min · the number you broughtThe number that took a model or a complicated method to produce, from your own work. Five passes, and at the end of it you will have two answers to the same question sitting next to each other.
Name the question. Write the question the number was actually answering, in the four slots: outcome, population, timeframe, decision. Not the question the method was designed for. The one somebody asked. If the decision slot is empty, ask the person today, because everything below depends on it.
Find the rung it used. Which of the five did the method sit on? A forecast, a score, a segmentation or a significance claim is rung 5. A year-on-year breakdown by region is rung 4. Be honest about this; the point is not to make the method sound modest.
Find the lowest rung that works. Read the question you wrote in pass one and go up from rung 1 until a rung answers it. Usually this lands one or two below where the method sat. Write which rung, and write the exact calculation: what is counted, and what it is divided by.
Compute it. Actually do the arithmetic, tonight, in a spreadsheet. This step is not optional and it is not a thought experiment, because the interesting part is nearly always how long it took, and you cannot estimate that afterwards.
Compare the two answers. Put them side by side and ask the only question that matters: would the decision have been different? Write the sentence either way. "The same call, in eleven minutes instead of three days" is a finding you can send to your manager tonight.
Use the question, not the method, to pick the rung. Methods do not have questions attached to them. They have defaults, and the default is whatever the person who wrote the tutorial was doing.
Write the denominator down explicitly before you compute anything. Per customer, per order, per opportunity. If you cannot say which, you are not ready to compute and the rest of the evening will be wasted.
Time yourself on the simple version. The number that persuades people is not the answer, it is the eleven minutes. Nobody argues with eleven minutes.
Keep the two answers in the same place. A demotion nobody can see is not a demotion. One line, both numbers, the decision in each case.
Check the mix before you trust the simple version. Session 4's four-word question still applies: did the mix change? A simple answer is not exempt from being a blend.
Do not go back and tell the original author they wasted their time. Often it was you. The output is a habit for next time, not an audit of last quarter.
Finding that the elaborate method was needed is a good outcome. It means the question genuinely was about many things at once, or genuinely was about confidence, and now you have written down why. Keep that sentence. It is the thing you produce the next time somebody asks why this took three days, and it is far more convincing written before the work than reconstructed after it.
Before session 6
25 min- Demote a second number, and this time note how long the simple version took, to the minute. Two examples is where it stops feeling like a trick and starts feeling like a rate.
- Find one count in circulation that has no denominator, on a dashboard tile, in a weekly email, on a slide, and supply one. Then check whether the story it tells is the same story the bare count told.
- Find one report where a more precise answer would not have changed any decision. Write the decision it fed and the two candidate answers that lead to it. These are common and nobody flags them, because flagging them is awkward.
- Bring a chart or a slide somebody else made, ideally one you were expected to accept without much scrutiny. Session 6 reads one in front of everybody, in the order worth asking the questions.
Three questions
Clear all three and session 5 gets stamped in your passport.
1 · How far up the ladder should you climb on any given question?
The rung above never answers the question below better; it answers a different question, more slowly. You can only tell which rung you need if the question has been written down with its outcome, population, timeframe and decision, which is why session 1 comes first. Climbing past that point costs time and buys nothing.
2 · Why is a bare count rarely an answer on its own?
An answer has to be comparable with something, and a bare count is comparable with nothing because the thing it is counting out of was left unstated. Complaints rising forty percent while orders rise sixty percent is the standard case: the count went up, the rate went down. Stating which denominator you used is part of stating the answer.
3 · Which pair of questions tells you it is time to stop climbing?
Both can be answered before the work starts. If the decision slot says the same thing at 12 percent and at 18 percent, narrowing the range is decoration. And an answer the asker can verify gets trusted and acted on, where one they cannot gets accepted politely and quietly ignored.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 5 cheat sheet · pin this
Everything so far has been about work you do: the question you were sent, the data you would need, the rows you looked at, the number you checked, the rung you stopped on. Session 6 is about work somebody hands you. The questions that find the weak spot in any chart, slide or deck, in the order worth asking them, and a final scorecard for the whole course. Bring the chart you did not make.