Three questions, three rungs

When somebody asks whether the email worked, they are asking about a world that does not exist: the one where the same customers, in the same winter, with the same tenure, did not receive it. Nobody has data from that world. Every number on the slide comes from the world where some people got the email and some did not, and those two groups were never the same kind of people to begin with.

That is not a data quality problem and it does not get better with a bigger warehouse. It is a difference in the kind of question being asked. Some questions are about what tends to appear alongside what, and a table answers those completely. Some questions are about what would change if somebody acted, and a table cannot answer those on its own, no matter how clean it is. Telling the two apart takes about fifteen seconds once you know the shape, and it is the highest-leverage fifteen seconds in a data review.

• Association. Two things move together in the data. True, measurable, and silent about why.
• Intervention. Somebody acts. The question is what changes as a result of the acting, not as a result of who tends to be acted upon.
• Counterfactual. The world that did not happen. What this particular customer would have done if the email had never arrived.
• Confounder. Something that causes both the action and the outcome. Winter causes more email and more spending, so winter is one.
• Holding fixed. Comparing like with like: December buyers against December buyers, five-year accounts against five-year accounts. The whole craft is knowing what belongs on that list.
• Estimand. The quantity you actually want a number for, written down in one sentence before anybody opens a tool. Most disputes about a result are really disputes about this, discovered late.
• Bias against noise. Noise is being unsure of the right answer. Bias is being sure of the wrong one. More data fixes only the first.

Rung 1 is the world of observed regularities. The questions here are about the joint distribution of things as they occur: how often do these two appear together, what is the average outcome among the people who did this, what is the rate in this segment against that one. Every one of them is answerable from a table, and answerable exactly, given enough rows.

This rung is not a lesser one. Almost all reporting lives here and it should: monitoring, segmentation, forecasting from patterns that keep holding, spotting that something changed. The mistake is never in asking a rung-1 question. The mistake is in accepting a rung-1 answer to a question that was asked on rung 2.

• Kestrelmark's rung-1 question. "Among customers who received the email, what was the average spend, compared with those who did not?" Answer: 10.54 units higher. Correct and reproducible.
• What it does not say. Anything about what would happen if the email were sent to someone new. The 10.54 describes two groups of people, not one action.

Rung 2 asks for the probability of an outcome when a variable is forced to a value, rather than observed at that value. The difference between forcing and observing is the whole discipline. When you observe that a customer received the email, you also learn something about the kind of customer they are: probably a long-standing one, probably one the campaign calendar picked in December. When you force the email on everybody, you learn nothing about them, because the choice was yours and not the calendar's.

Almost every question that carries a budget is on this rung. Should we send it. Should we build it. Should we change the price. Should we keep paying for this channel. Each of them is a question about acting, and each of them requires a statement about what the action changes and what it leaves alone. That statement is a model of the situation, and it has to come from people who know the business, because no column in the warehouse contains it.

Rung 3 asks about a world that did not happen, for a case that did. Not "what happens on average if we send the email" but "this customer received it and then bought; would she have bought without it?" That requires holding two worlds side by side for the same individual, only one of which was ever observed, and it can only be done from a fully specified model of how the outcome is produced.

This is the rung of blame, credit, and attribution, and it is why those conversations are so hard. "The campaign saved this account" is a rung-3 claim. So is "we would have hit the number anyway". Both are answerable in principle and neither is answerable from a comparison of groups.

• Where leaders meet rung 3. Retention saves, churn prevention, incrementality of a discount to a customer who was going to buy, and any sentence containing the word "because" about a single case.
• What it costs. Everything rung 2 needs, plus a specification detailed enough to run the same individual through both worlds. Practitioner session 6 does this properly for Kestrelmark.

This is the formal result underneath the whole evening, and it is worth stating as flatly as it deserves. The information that lives on rung 1 - the joint distribution of everything you observed, in unlimited quantity - is not sufficient to answer a rung-2 question. Not hard to use, not noisy: insufficient. The same table is consistent with the email causing the spend, with high spenders being the ones selected for the email, and with December causing both. Choosing between those three requires something that is not in the table.

The practical consequence is uncomfortable and worth saying out loud in your next review. A dashboard cannot answer "did the email work", no matter how much data is behind it. Not a better dashboard, not a real-time one, not one with more dimensions. The limitation is in the kind of information a dashboard holds, not in its size or its freshness. What closes the gap is a statement about which things cause which - and that statement comes from the people who know how the campaign was targeted.

The instinct when a number is questioned is to get more of it: a longer window, a bigger sample, the full base instead of a slice. That instinct is exactly right for one problem and exactly wrong for the other, and the two look identical on a slide.

• Noise is the wobble around the right answer. It shrinks as the sample grows, which is why more data feels like the universal fix.
• Bias is a systematic offset from the right answer. It does not shrink. Kestrelmark at 5,000 customers gives 10.54; Kestrelmark at a million customers gives 10.54 with much tighter error bars, which is a more confident wrong answer and a more persuasive slide.
• The confidence interval is silent about this. It describes how much the estimate would move if you drew a different sample from the same flawed comparison. It has nothing to say about whether the comparison was the right one.

So a leader hearing "the sample was 900,000 customers" has learned that the error bars are small. They have learned nothing at all about whether the number is the effect of anything.

No tooling, no notice, about ninety seconds. Ask them in order and stop at the first one that has no good answer, because the later questions do not matter until the earlier ones are settled.

1. Which rung is this question on? Is somebody going to act on the answer by doing something differently? Then it is rung 2, whatever the slide says. If the answer only informs a description of what is happening, it is rung 1 and the reporting is appropriate.
2. What would we have to hold fixed for this comparison to be fair? Name the things that push both who gets treated and what the outcome is. At Kestrelmark that list was two items long and took a minute to produce, from people who knew the campaign calendar.
3. Do we have those things? Not "could we get them eventually": are they columns we can use for this analysis. If the list from question two contains something nobody measures, that is the finding, and it is a much more useful finding at the start than after a month.

What makes this work in a room is that all three questions are about the business rather than about statistics, so the people who can answer them are already at the table. Nobody has to defend a method. They have to say what drives who gets the email, which is something a campaign manager knows off the top of her head.

This is the single most common mismatch in a data team's output, and it is nobody's fault in particular. Reporting infrastructure is built to describe what happened, because that is what can be built reliably and refreshed nightly. Decisions are about what to do next. The two are one rung apart and the seam between them is usually invisible, because the same numbers appear on both sides of it.

• The symptom on the team's side. Analysts producing accurate work that keeps getting questioned, and not understanding why "but is it causal" arrives after every delivery. They answered the question they were given.
• The symptom on the leader's side. A vague sense that the numbers are right but the decisions do not get easier. That feeling is the rung gap and it does not close by asking for more dashboards.
• The cheap fix. Label the rung when the question is commissioned, not when the result is reviewed. One word in the request, and both sides know what would count as an answer.
• The honest caveat. Labelling a question rung 2 does not make it answerable. Sometimes the correct outcome of the triage is "we cannot answer this from what we have, and here is what we would have to run instead". That is a good day, arriving early.

When the answer is that the data cannot support the claim, the next question is which method could - and estimating an effect from observational data belongs to learn-experimentation-with-phoebe, which covers potential outcomes, propensity scores and the rest of that machinery in working depth.