Not everyone responds

The number that comes back from a causal analysis is almost always an average across everybody in the study. It answers a question of the form "if we had done this to all of them instead of none of them, how much more spend would there have been". That is a real question and it is worth answering carefully, which is what the first four sessions were about.

A rollout is a different question. It asks who to send the thing to, at what cost, and the answer depends on how the effect is spread across the population rather than on where its centre sits. A campaign can have a solidly positive average and still be the wrong campaign, because the average is silent on whether the gain sits in a tenth of the base or in all of it, and silent on whether anybody was made worse off.

• Persuadables - they act because the treatment reached them, and would not have otherwise. This is the entire source of value in the campaign, and every unit of budget that lands anywhere else is waste.
• Sure things - they were going to act either way. Reaching them costs money and produces nothing, though it produces a very healthy-looking response rate, which is why response rate is the metric that keeps bad campaigns alive.
• Lost causes - they were never going to act, treated or not. Cheap to be wrong about, since the only loss is the cost of the send.
• Sleeping dogs - the treatment makes them less likely to act. They unsubscribe, they mark the sender as spam, they remember they were meaning to cancel. Reaching them costs money and destroys value on top.

These names come from marketing practice and they are not technical terms, but the split behind them is exact: for a yes-or-no treatment and a yes-or-no outcome, every person falls in one of those four boxes and no other. A leader who can ask "what share of the file is in each box" has asked the only targeting question that matters.

For each customer there are two possible spend figures: the one that follows from receiving the email and the one that follows from not receiving it. Their difference is that customer's own effect. Exactly one of the two ever happens, so no person's effect is ever measured. This is not a data-collection failure that a better warehouse would fix; it is the shape of the problem.

What a well-identified analysis recovers is the average of those unobserved differences over a group. It is a genuine quantity, it is the right answer to the question "does this work", and it has one property worth stating out loud in every readout: the average need not equal any individual value. A household with 2.4 children does not exist either, and nobody has ever tried to design a car seat for one.

• An average over everybody answers a rollout-to-everybody question. If the plan is to treat the whole base, the average really is the number you want, and the cost side below is what decides it.
• An average over a segment answers a segment question. Which is why the useful ask is not "the effect", but "the effect for whom".
• An average hides its own composition. Two campaigns with identical averages can differ in every way that a budget owner cares about.

Take a treatment that is either applied or not, and an outcome that either happens or does not. Every person in the file is in one of four boxes: they act only if treated, they act either way, they act neither way, or they act only if left alone. That exhausts the possibilities, and the four boxes behave completely differently under a budget.

• Persuadables are the whole business case. The incremental revenue of the campaign is created here and nowhere else. At Kestrelmark they are 20.4 percent of the file, gaining 29.09 each, so the large majority of the send budget lands somewhere that produces nothing or worse.
• Sure things are the flattering group. They open, they click, they buy, and they would have bought regardless. At Kestrelmark they are 22.4 percent of the file and the email is worth 2.10 to them. Every response-rate dashboard rewards reaching more of them, and every one of those responses is a cost with no return behind it.
• Lost causes are cheap mistakes. The loss is the send cost, nothing more. They are the largest group here, 49.3 percent, at 1.04 each. In an email programme the send is close to free, which is precisely why email programmes grow until the persuadable share is tiny.
• Sleeping dogs are the expensive mistake. Contact triggers the unsubscribe, the spam complaint, the cancellation the customer had been putting off. Kestrelmark's are the 7.9 percent who have cancelled once already, and the email costs them 8.87 each. The cost is the send plus the destroyed value plus a channel that no longer reaches them.

The uncomfortable consequence: two campaigns can post the same average effect while one of them is mostly persuadables at a modest gain and the other is a large persuadable gain partly cancelled by a group being driven away. Kestrelmark is the second kind, and now measurably so. That is a better campaign badly aimed, and no summary metric tells it apart from the first kind until somebody splits the file.

Work this one slowly, because the composition is the argument. 5,000 customers, a true average effect of 6.23, and four segments that the file can already tell apart.

• The gain is concentrated. 20.4 percent of the file, 1,022 people, gain 29.09 each. Essentially all of the value the campaign creates is created here.
• Two segments barely move. 22.4 percent gain 2.10 each and 49.3 percent gain 1.04 each. Between them they are most of the base, and the email is close to a rounding error for all of them.
• One segment is harmed. 7.9 percent, 393 people, lose 8.87 each. These are accounts that cancelled once and were won back, and the monthly reminder that a subscription exists is doing them measurable damage.

Now read the headline again. A campaign that reached only the 1,022 would be worth 29.09 per head instead of 6.23, and would stop paying to damage 393 customers a month. Nothing in the average distinguishes this world from one where nearly everybody gains a little and nobody is harmed at all, and nothing in the standard readout would surface the difference. The average is not wrong. It is silent, and silence was read as reassurance.

The Kestrelmark work is unusually clean by the time it reaches this point. The naive comparison of emailed against not emailed gives 10.54. Holding season and tenure fixed brings it to 6.56, against a truth of 6.23, so the identification work has done its job and the gap left over is 0.33. That is a good analysis by any standard the first four sessions set.

And on the strength of that single number the email went to the entire subscriber file, monthly. The step from 6.23 to "send it to everybody" was never argued; it was assumed, because the number was positive and there was only one number. Three separate questions were skipped in that step, and all three have since been answered from the pilot data that was already sitting there.

• Who produced the 6.23. Customers who lapsed 30 to 90 days ago, 20.4 percent of the file, at 29.09 each. The rollout to the whole file spends the large majority of its budget outside the only group that earns.
• Whether anybody was moved the other way. Yes: the 7.9 percent who have cancelled once already lose 8.87 each. They were the textbook sleeping-dog population, and the campaign was aimed at exactly them.
• What the send costs. 7.00 fully loaded, against segment effects of 29.09, 2.10, 1.04 and -8.87. One of those four clears the cost. Part 2 walks the arithmetic.

None of the three required a new dataset. Two were a segment split of an analysis that already existed, and the third was one number from finance.

Listen to how the request arrives. "Should we roll the email out?" "Is this worth the budget next year?" "Do we keep doing this?" Every one of those is a policy question: it asks for a rule about who gets treated. What gets written on the analysis ticket is "measure the incremental effect of the email", which is an effect question and returns a single average.

The gap is invisible because the average answers the policy question in one special case: when the policy under consideration is "treat everybody or treat nobody" and there is no cost per unit. That case is rare, and it is nobody's actual choice, because the real menu always includes "treat this segment" and there is always a cost.

• "Does it work" needs a centre. One number, one interval, one comparison against zero.
• "Who should get it" needs a spread. Which groups gain, by how much, and which lose. A ranking, not a point.
• The second cannot be derived from the first. No amount of precision on the average recovers the distribution behind it, which is why asking for a tighter confidence interval is answering the wrong request more carefully.

Every send has a cost: creative, deliverability, list health, the attention it spends, and in most channels a real cash number per unit. Kestrelmark's fully loaded cost per treated customer is 7.00 units against the same currency as spend. The campaign's average effect is 6.23. Put the two beside each other segment by segment and the programme stops looking like a modest success.

One segment out of four earns its send. Everything else on the list is paid for and returns less than it cost, and the smallest group returns less than nothing. That turns into two numbers a budget owner can act on.

Same intervention, same true effect of 6.23, same correctly identified analysis. The difference between -3,848 and +22,578 is entirely a question of who was on the list. This is the sharpest form of tonight's argument: the programme as run destroys value, and the number that authorised it is a healthy-looking positive 6.23.

• The rule is one line and nobody applies it. A segment is worth sending to when its own effect exceeds the cost per send. At 7.00, exactly one of Kestrelmark's four segments passes, and it is a fifth of the file. No model is needed to check this, only the effect per segment and a number from finance.
• Perfect targeting is not on the menu. The +22,578 row assumes the persuadable segment can be reached exactly. Recency is a real field in the warehouse rather than a score, so that row is closer to achievable here than it usually is, but a four-bucket segment is still coarse and every bucket contains people who do not behave like their label. Treat it as the ceiling and ask how much of the gap a crude segment closes.
• The cheaper the channel, the worse the discipline. When the per-unit cost rounds to nothing, the break-even share rounds to nothing too, and the programme expands until the sleeping dogs dominate. The cost that eventually bites is the channel itself.

Having argued that the distribution is what matters, the fair thing to say next is that the distribution is much harder to recover than the average. An average pools the whole sample; an effect for a subgroup uses only that subgroup, and an effect for a person uses a sample of one, which is why individual effects are estimated rather than observed and are estimated badly.

This is a statistical fact and not a reason to give up, but it does set the shape of a sensible ask. A model that assigns every customer a personal uplift score will produce a beautifully ranked list, and much of that ranking will be noise that reverses when the model is refitted next month. A split into four or five segments defined by things the business already understands will be coarser and far more stable. Kestrelmark is the cheerful version of this: the split that mattered turned out to be a single field the business had all along, four recency buckets that anyone in the room can describe.

• Ask for few, large, interpretable segments. Recency band, tenure band, plan tier, channel. If a segment cannot be described in a sentence, it cannot be reviewed, and an unreviewable targeting rule is a permanent unaudited policy.
• Ask whether the split was pre-specified. Segments discovered after seeing the results will always contain a spectacular one. Discovery is legitimate as a source of hypotheses and illegitimate as a source of conclusions.
• Ask what happens if the ranking is wrong. A targeting rule that loses little when the ranking is mediocre is worth having. One that only pays off when the ranking is excellent is a bet on the model, not on the campaign.

The methods that estimate effects conditional on customer characteristics, and that turn those estimates into a treat-or-not rule, are worked properly on the practitioner track rather than named vaguely here.

• Practitioner session 7 covers conditional average effects and the three standard meta-learners, each with the weakness its authors documented: the single-model approach whose regularisation pulls the effect toward zero and can drop a weak effect entirely, the two-model approach that avoids that and still pays a regularisation cost, and the approach built for the treated-to-control imbalance that is normal outside experiments.
• Practitioner session 8 covers targeting: ranking by predicted uplift, reading the gain against a random-targeting baseline, and choosing where on that curve to stop sending. That is where the coarse-segment discipline above gets a number attached to it.
• Estimating any of this from observational data is a different course. Propensity scores, inverse probability weighting, doubly robust estimation, difference-in-differences, synthetic control and the DoWhy workflow all belong to learn-experimentation-with-phoebe, which teaches each of them properly. The work here stops at what the question is, whether it can be answered, and who the answer is about.