Counterfactuals: the world that did not happen

A rung-2 question fixes a variable by force and asks about the resulting distribution: send the email to all 5,000 customers, send it to none, difference the mean spend, and the answer is 6.23. Nothing in that question refers to a particular person. It is a statement about the population under a policy, and the whole of sessions 1 to 5 was about earning it.

A rung-3 question does refer to a particular person, and it refers to her twice. It says: take Amara exactly as she is, with the email she actually received and the spend she actually recorded, and hold everything about her fixed except the treatment. Now report her spend in the other world. Two worlds are being described at once, and one of them left no data behind, which is why the question cannot be phrased as a comparison between rows.

• Rung 1. "Customers who got the email spent 10.54 more." True, reproducible, and mostly season and tenure. Session 1.
• Rung 2. "Sending the email to everyone raises mean spend by 6.23." Needs a graph, an adjustment set that closes the back doors, and the data to hold it fixed. Sessions 2 and 3.
• Rung 3. "Amara would have spent 31.0 less had we not sent it." Needs everything above, plus the functional form of every equation in the model.

The three sentences use the same words and cost wildly different amounts to defend. A useful habit is to write the rung number in the margin of any claim before checking anything else about it, because a rung-3 sentence backed by a rung-2 analysis is the most common way a defensible number turns into an indefensible one.

For rung 2 the graph carries all the assumptions that matter. It tells you which paths are causal, which are back doors and which set closes them, and once a valid set exists the arithmetic can be anything: a group mean, a regression, a stratified average. The shape of the equations is not part of the question.

For rung 3 it is the entire question. To hold Amara fixed you have to know what "Amara" consists of beyond her observed columns, and what she consists of is the collection of error terms that made her row come out the way it did rather than at the average for her covariates. Those errors are only recoverable by inverting the equations that generated the row, so the equations have to exist and be written down.

• The graph gives you the inputs to each equation. It says spend is a function of season, tenure, email, opened and something unmeasured. It does not say the function is linear, or additive, or monotone.
• The functional form gives you the residual. Subtracting the model's prediction for her covariates from her actual spend is what isolates the part of her that is hers.
• The flipped world is then a re-run, not an estimate. Once her errors are pinned, setting the email to 0 and recomputing is deterministic arithmetic with no sampling in it.

This is exactly what the bench's engine does when it produces the number 6.23: it runs the written model twice over the same 5,000 draws, forcing the email on and then off, and differences the mean. The truth is available there only because the model is available. Outside a simulator it is not, and that gap is the honest cost of rung 3.

• Recover. Take the observed row and solve each structural equation for its own noise term. Kestrelmark's spend equation is additive, so her spend minus the model's prediction at her own values is her residual, exactly.
• Flip. Overwrite the treatment with the value you want to ask about, and change nothing else. Not her season, not her tenure, and above all not the noise you just recovered, because that noise is what makes this her rather than a customer like her.
• Recompute. Push the flipped row back through the equations in causal order. Downstream variables move: with the email off, her chance of opening drops from sigmoid(1.4) to sigmoid(-0.2), and if she stops opening then the mediated 3.0 goes with it, on top of whatever her segment's direct effect was worth.

Kestrelmark's spend equation gives opening a coefficient of 3.0 for everybody, and gives the email a direct coefficient that depends on which recency segment the customer sits in: 1.0 for an active "sure thing", 28.0 for a lapsed "persuadable", 0.0 for a dormant "lost cause", and -10.0 for a "sleeping dog" who has cancelled once before. So flipping one customer's email off can still do exactly two things, and her segment decides what each of them is worth.

• She would have opened anyway, or she never opens. The open flag is unchanged, only the direct term drops, and her individual effect is her segment's direct effect. Amara is a persuadable, so hers is 28.0.
• The email is what made her open. Both terms drop, and her individual effect is her segment's direct effect plus 3.0, which for Amara is 31.0.

Four segments, two branches each, and the population holds exactly the eight values that fall out of it: -10.0 and -7.0 for the sleeping dogs, 0.0 and 3.0 for the lost causes, 1.0 and 4.0 for the sure things, 28.0 and 31.0 for the persuadables. The average of those eight is 6.23, and 6.23 is not among them.

Read the average the other way round and it decomposes cleanly: a mean direct effect of 5.16, plus 3.0 times the change in open rate the email causes, a mediated 1.056, giving an analytic 6.22 against the bench's brute-forced 6.23. The two agree, which is the engine checking itself against the algebra rather than against its own output.

That eight-point distribution is real heterogeneity rather than noise, and it is what session 7 tries to recover with a model instead of with the equations.

Three shapes of question genuinely require an individual counterfactual, and they are worth recognising because each one is usually asked in language that hides it.

• "Would this customer have churned anyway?" The retention team wants to know whether the save was caused by the intervention or would have happened without it. That is her outcome in a world where nothing was sent, and it is the attribution question underneath every save-rate report.
• "Was this particular sale caused by the campaign?" A question about one transaction that already happened, sometimes with money attached to the answer. The formal version is the probability that the outcome would have been different had the treatment not occurred.
• "Is this individual a good person to treat next time?" Forward looking, and therefore the friendliest of the three, because it can be softened into a question about people like her rather than about her. That softening is what session 7 does.

Notice that the first two are retrospective and about a specific event, which is what makes them expensive. There is no future experiment that settles them.

Two models can fit the observed rows equally well, agree on every prediction you can check, and still give different counterfactuals for the same person. This is not a small-sample problem and it does not shrink with more data. The observed rows constrain the model only where the rows are, and a counterfactual asks about a point where, by definition, there is no row.

• The residuals are identical, so the usual diagnostics are silent. There is no held-out set for a world that did not happen, and cross-validation scores the same predictions both forms already agree on.
• The disagreement is asymmetric. Averaged over the population the two forms tend to agree closely, because their extrapolation errors point in different directions for different people and cancel. One individual has nothing to cancel against.
• A binary variable makes it worse. Amara's open flag pins her open-equation noise to an interval rather than a point, so even with the true equations her counterfactual is bounded rather than determined. Partial identification is the normal case, not an edge case.

Before paying for rung 3, check that the decision actually needs it. The test is whether the action attaches to a person or to a policy. "Should we send this campaign" is a policy and rung 2 answers it with an average. "Which segment should receive it" is still a policy, one per segment, and rung 2 answers that too by conditioning. Only "what happened to this account" is unavoidably rung 3.

• Reaching for rung 3 unnecessarily adds a functional form to your assumption list, which is the one assumption no reviewer can check and no sensitivity analysis in session 9 will price for you.
• It also invites individual-level reporting, and a number attached to a named customer is treated as a fact by everybody downstream, regardless of the range it came from.
• The intermediate position is a conditional average: the effect for customers like her rather than for her. It needs no structural form beyond what rung 2 already required, and it is what session 7 estimates.

A conditional average treatment effect is the per-person counterfactual with the individual sanded off. Instead of asking for Amara's difference, it asks for the average difference among customers who share her observed covariates, which is a quantity that averages away the residual you would otherwise have to recover.

• It needs no structural equations. Whatever identified the average effect in session 3 identifies the conditional one within each covariate cell, so the assumption list does not grow.
• It gives up the individual. Two persuadables with identical covariates get identical answers, even though the truth here says one of them gets 28.0 and the other 31.0.
• It is what the business usually wanted. Targeting acts on segments, and a segment is exactly what a conditional average describes.

So the honest ordering is: use rung 2 where the decision is a policy, use a conditional average where the policy varies by segment, and reserve rung 3 for the retrospective questions that genuinely name an individual. Session 7 builds the middle one, and gets remarkably far: the recency segment on its own reaches an R^2 of 0.986 against the individual effect. What is left over is exactly the 3.0 coin flip, and no feature in the table predicts it. That is the honest statement of what rung 3 gives you here. A fully specified model buys you the segment. It never buys you the person.