Simpson's paradox

The shape has a precise definition. An association between two variables holds in every level of a third variable, and reverses when that third variable is ignored. It also runs the other way round: no association in the aggregate, a strong one in every subgroup. Both are the same phenomenon and both are ordinary arithmetic.

What makes it worth a session is the frequency. Any time a treatment is allocated unevenly across groups that differ in baseline outcome, the ingredients are present. Uneven allocation is the normal state of affairs outside an experiment, because somebody chose who to treat and they chose for a reason.

• Stratum - one level of the variable you split on. Two strata tonight; the effect is identical with twenty.
• Aggregate - the comparison with the stratifier ignored, which is the same thing as pooling the strata weighted by their own row counts.
• Reversal - the sign of the comparison differs between every stratum and the pool.
• Unequal allocation - the treated and untreated groups have different proportions in each stratum. This is the engine of the reversal; with equal allocation it cannot happen.
• Berkson's paradox - the collider-driven cousin, where selecting on a common effect creates an association between two unrelated things. Session 2.
• Stratifier role - confounder, mediator, collider, or bystander. The whole resolution turns on this one word, and the word comes from the graph.

Four divisions, each exact. Routine offers, 80 of 100, is 80.0 percent. Routine no-offers, 600 of 800, is 75.0 percent. At-risk offers, 240 of 800, is 30.0 percent. At-risk no-offers, 25 of 100, is 25.0 percent. The offer is ahead by exactly five points in both strata.

Now pool. The offer column: 80 plus 240 renewals is 320, out of 100 plus 800 accounts, which is 900. That is 35.6 percent. The no-offer column: 600 plus 25 renewals is 625, out of 800 plus 100 accounts, also 900. That is 69.4 percent. Same 1,800 accounts, same 945 renewals, and the winner has changed.

The reason is visible in the counts and nowhere else. The offer group is 89 percent at-risk accounts, whose ceiling is low. The no-offer group is 89 percent routine accounts, whose floor is high. Pooling compares a mostly-hard sample against a mostly-easy one, and the five-point advantage inside each stratum is buried by a forty-five-point difference in stratum composition.

A paradox is a contradiction. There is no contradiction here: two different quantities were computed and they came out different, which is what different quantities do. The stratified comparison answers "among accounts like this one, does the offer help". The pooled comparison answers "of the accounts that got the offer and the accounts that did not, which group renewed more". Those are separate questions with separate answers and both answers are right.

The surprise comes from an unstated assumption that one of them is the effect of the offer. Neither is, automatically. A comparison becomes an effect only when the groups being compared differ in nothing but the treatment, and whether that holds is a claim about how accounts came to be treated, which lives outside the table.

• The arithmetic is never in dispute. Do not go looking for the bug. There is no bug, and hunting for one wastes the afternoon that should have been spent on the graph.
• The dispute is about which comparison is the causal one. That is a modelling question, and it has an answer, and the answer is not in the data.
• Both numbers can be reported honestly. What cannot be done honestly is reporting one of them as the effect without saying why the other was rejected.

• Any targeted campaign. Targeting is deliberate unequal allocation, so the ingredients are present by design in every campaign readout that was not randomised.
• Any triage. Treatments given preferentially to severe cases, support tickets routed by difficulty, remedial teaching given to struggling students. The treated group is worse off before anything happens.
• Any comparison across time. If the mix of customers changed between two periods, the aggregate can move in the opposite direction to every segment within it.
• Any weighted average that somebody unweighted. Pooling raw counts silently reweights by group size, and group size is rarely a quantity anybody chose on purpose.
• The tell is always the same. Look at the allocation, not the outcome. If the treated and untreated groups have very different compositions, the aggregate is a comparison of compositions wearing a treatment label.

Session 2 named the shapes. Applied to a stratifier, each one gives a different answer to "should I split on this".

• Confounder, a common cause of treatment and outcome. Split. The stratified numbers are the causal ones and the pooled figure is contaminated by the back-door path. In the renewal table, if the churn risk both determined who got the offer and predicted renewal on its own, this is the case: the offer helps, by five points, in both strata.
• Mediator, on the causal path from treatment to outcome. Do not split. Holding it fixed removes part of the effect you are trying to measure, so the stratified numbers understate and the pooled figure is closer to the total effect. Session 10 measures exactly this on Kestrelmark's opened variable, at 5.48 against a total of 6.23.
• Collider, a common effect of treatment and outcome or of their causes. Do not split. Conditioning on it manufactures an association that was not there, so the stratified numbers are worse than the pooled ones. Session 10 prices Kestrelmark's two colliders at 0.97 for the loyalty score, which precedes the email, and 3.64 for the ticket flag, which the email helps cause: the downstream one is by far the more expensive of the pair.
• Bystander, connected to neither. Splitting changes nothing at all. Kestrelmark's device variable moves the estimate by nothing to two decimal places, which is the cleanest possible reminder that harmful variables cannot be spotted by feel.

The uncomfortable part is that the table looks identical in all four cases. There is no column of the data, no correlation, no fit statistic that distinguishes them. The distinguishing information is the direction of the arrows, and arrows come from knowing the business.

The re-engagement email has the same ingredients. The winter campaign calendar sends more email, and people buy more in winter regardless of email, so season both allocates the treatment and drives the outcome. Long-tenured accounts sit on more lists and spend more anyway, so tenure does the same. Two confounders, two back doors, unequal allocation across both.

The aggregate comparison of emailed against not emailed gives 10.54. Holding season and tenure fixed gives 6.56, against a true effect of 6.23 known by construction from the model that generated the customers. The aggregate overstates by 4.31 units, which is two thirds of the true effect again.

Both roles are confounders here, so the verdict is unambiguous: 6.56 is the number to act on and 10.54 is the number to explain. Note how little the size of the gap tells you on its own. A gap of that size under a mediator would have meant the opposite, and no amount of staring at 10.54 and 6.56 would reveal which case you were in.

Both rules of thumb are attempts to replace a causal judgement with a procedure, and neither can work, because the two cases they are trying to cover recommend opposite actions on identical tables.

• "Always segment" fails on mediators and colliders. Segmenting by something the treatment caused answers a different question at best and manufactures bias at worst. The habit is popular because segmenting almost always produces an interesting-looking difference.
• "Never trust the aggregate" fails on mediators too, and on any case where the pooled figure was the total effect all along. It also has no stopping rule: there is always one more variable to segment by, and segmenting far enough reaches subgroups of one.
• The replacement rule is short. Name the role of the variable in the graph before splitting on it. If nobody can name the role, that is the finding, and the split waits.
• Berkson's paradox is the same lesson in the collider case. Selecting on a common effect creates an association between causes that were independent. Session 2 built it; here it is one of the four verdicts.