The four answers bench

Levels 2 and 3 split the data and look inside each piece. That catches a lot, and it is what most people mean by "did you break it down". What it does not do is put the pieces back together.

Level 4 does. It asks: if the mix of the groups had stayed exactly as it was before, what would the after number be? You take the after period, compute the figure inside each group, and then average those using the before period's group proportions. Nothing else changes. If that number disagrees with the headline, the difference was never about the thing you measured. It was about who showed up.

This matters because a change that shifts the mix is extremely common and almost never announced. A promotion pulls in a different kind of customer. A new region opens. A product launch changes what people buy. In every one of those cases the headline is a blend of two things moving at once, and the blend can point the opposite way to both of its parts.

Level 1 is two minutes and you should always do it, because it is the number everybody else already has in their head and you need to know what you are contradicting.

Levels 2 and 3 are ten minutes each if the grouping columns exist, and impossible if they do not. That is not a small caveat: whether you can do this at all was decided by whoever designed the data collection, which is why session 2 asks what you would need before you look at what you have.

Level 4 is another ten minutes and one judgement call, which is which mix to hold fixed. Hold the wrong one and you have done careful arithmetic on an irrelevant question. There is no formula for picking it; the honest rule is to hold fixed whatever changed for reasons unrelated to the thing you are measuring. In tonight's data that is the channel mix, because the promotion deliberately pushed traffic through one channel.

Revenue per order, headline: up thirteen pence. If you stopped there you would write "revenue per order held steady through the promotion" and nobody in the room would challenge it. It is correct.

Split by channel: email orders fell by GBP 3.83, social orders fell by GBP 2.97. Both down, substantially. Also correct.

Those two facts are not in conflict, and understanding why is the single most useful thing you can take from a session like this. Social orders are worth more than email orders. The promotion pushed the channel mix from roughly thirty percent social to roughly sixty percent. So after the promotion, more of the orders came from the expensive channel. Each channel got worse; the blend got very slightly better, because its composition improved faster than its parts deteriorated.

The like-for-like number settles it: hold the mix where it was and revenue per order fell by GBP 3.57. That is the number that answers the question "did the promotion make orders less valuable", and it is the opposite of the headline.

Notice also which split found it. Splitting by customer type showed new down GBP 0.82 and returning up GBP 0.59, which looks like noise and would have reassured you. Only the channel split revealed it. That is why level 3 exists: the first grouping you try is not always the one that carries the story.

Conversion rose 2.9 points in the headline, 2.0 and 3.5 points inside the two channels, and 2.5 points like-for-like. Every level agrees. The two-minute answer was the correct answer, and the extra thirty minutes of care bought nothing except the knowledge that it bought nothing.

That is not a waste, and it is worth saying clearly because the opposite lesson is easy to take from a session like this. The four levels are not a tax you pay on every number forever. They are how you find out whether this particular number is one you can quote. Sometimes the answer is yes in ten minutes, and then you stop and go and do something else.

What you cannot do is skip the check because the number looks reasonable. Both of tonight's answerable questions looked entirely reasonable at level 1.

Take any before-and-after figure from your own work. Conversion, response rate, average order value, time to resolve, headcount cost, anything with a before and an after.

Level 1: write the headline and its direction. Level 2: name the one grouping everybody would ask about first, and say whether you could compute it with the data that exists. Level 3: name a second grouping that nobody asks about, which is where these things usually hide. Level 4: ask whether the mix of those groups changed between the two periods. If it did, and you have not held it fixed, you do not yet know what your number means.

Most people find they can do levels 1 and 2, cannot do 3 because the column does not exist, and have never once done 4. That is a completely normal result and it is the honest state of most reporting.

The bench's dataset is constructed, so here is the mechanism happening to somebody real. In autumn 1973, graduate admissions at Berkeley looked like this in aggregate: 8,442 men applied and 44 percent were admitted; 4,321 women applied and 35 percent were admitted. A nine-point gap across 12,763 applicants, far too large to be chance.

Broken down by department, it mostly went away or reversed. In the largest department, 62 percent of male applicants were admitted against 82 percent of women. In the second largest, 63 against 68. Across the six biggest departments the aggregate still showed men ahead, 45 against 30, while the individual departments did not. Women were applying in much larger numbers to the departments that admitted almost nobody, and men to the ones that admitted most people.

What the authors actually concluded is more careful than the version that gets retold. They found four departments significantly biased against women and six significantly biased against men, and their pooled, corrected figure showed a small but statistically significant bias in favour of women. The conclusion did not rest on counting biased departments; it rested on pooling the rates while weighting by each department's own rejection rate.

Two honest notes. The original paper could not be retrieved during this build, so every figure above comes from secondary sources that reproduce its tables and agree with each other to the digit. And the story is usually told as "Berkeley was sued for sex bias", which appears to be false: the university's graduate division feared a lawsuit and commissioned the analysis, and no suit was filed. That version has been traced through at least six textbooks and papers. The most-cited example of a misleading aggregate comes wrapped in its own misleading story, which is a fitting thing to know before session 6.

The third question, "did it bring in customers who stay", is the one the founder actually cares about, and the dataset ends twelve weeks after the promotion with no record of what anybody did next. Every level returns the same thing: no column.

The professional move there is to say so, immediately and without apology, and to say what would answer it: a follow-up window of at least one renewal cycle, joined to the same customers. That is a two-sentence reply on the day the request arrives. The failure mode is to answer a nearby question instead, because a nearby question is answerable and the real one is not, and then to present that answer as though it were the one that was asked. Session 2's gap list exists to catch exactly this before any calculation starts.