What not to adjust for

A covariate list is usually treated as a technical detail, delegated downward, approved on the grounds that it looks thorough. It is not a technical detail. It is the place where the analysis quietly decides which question it is answering, and the decision is invisible in the output: every one of the sets below produces a clean regression, a plausible coefficient and a defensible-looking table.

The reason this belongs to whoever commissioned the work is that the information needed to judge the list is not in the data. It is knowledge about how the business actually operates - what happens first, what causes what, what the scoring model was built from. That knowledge sits with the people who run the process, not with the person writing the query.

• Confounder - something that causes both the treatment and the outcome. Season and tenure at Kestrelmark: winter sends more email and sells more; long-tenured accounts get more email and buy more anyway. These are the variables that must be adjusted for.
• Mediator - something on the causal route from treatment to outcome, carrying part of the real effect. Whether the customer opened the email. Adjusting for one is not an error; it changes the question.
• Collider - something caused by two other things at once. Whether a support ticket was raised, which is caused both by getting the email and by spending more. Adjusting for one manufactures an association that was not there.
• Bystander - something related to neither. The customer's device. Harmless, and worth knowing about because it proves the other three cannot be spotted by feel.

These four words are the whole vocabulary for tonight. A leader who can ask "which of these four is it" for every line in a covariate list has most of the value of the technique.

Kestrelmark's email works partly because people open it and then buy. Whether the customer opened is therefore on the causal route, carrying part of the effect the campaign is being credited with. Hold it fixed and the estimate falls from 6.56 to 5.48, a bias of 0.75 against the true total effect of 6.23.

Notice what 5.48 actually is. The direct part of the effect, averaged over the customers in the model that generated this data, is 5.16. The analyst who adjusted for opening has not miscalculated anything. She has measured "what does the email do for somebody who never opens it", accurately, and reported it under a heading that promised the total effect.

• Total effect answers "should we send the email". Nearly always the business question, and 6.23 is its answer.
• Direct effect answers "how much survives if nobody opens it". A real question for a subject-line project, and 5.16 is its answer.
• Nothing in the output distinguishes them. Both are a coefficient with a small standard error and a sensible sign.

A support ticket at Kestrelmark has two causes. Getting the email prompts questions, so the email raises the ticket rate. Spending more also prompts questions, so higher spend raises it too. Two arrows point into one box, and that box is a collider. Adjusting for it moves the estimate to 2.59, a bias of 3.64, and that is by a wide margin the largest single harm on this bench.

The rule behind this is worth stating in its general form, because it applies far outside analytics. In a structure i -> m <- j, conditioning on m makes i and j dependent when they were independent. Epidemiologists call it Berkson's paradox, after the observation that among hospital patients two unrelated diseases appear correlated: each one is a reason to be admitted, so knowing a patient is admitted and does not have the first raises the chance they have the second.

The same machinery runs at Kestrelmark. Among customers who raised a ticket, one who did not get the email probably spent a lot, because something had to prompt the ticket. Conditioning on the ticket has invented a relationship between the email and spend that the business never created, and it pushes the estimate down.

This is the one that surprises experienced teams. Kestrelmark's loyalty score is computed monthly, long before the campaign is decided, from behaviour that predates the email entirely. By the usual folk rule it is safe: it happened first, so it cannot have been affected by the treatment. Adjust for it and the estimate falls to 5.26, a bias of 0.97. That is the middle of tonight's three harms by size, and it is still bias, arriving from a variable the folk rule certified as safe.

The reason is the shape, not the clock. Two things nobody measured feed into that score. One of them is the disposition that also makes a customer a target for the campaign; the other is the disposition that also makes them a spender. The loyalty score is their common effect, so conditioning on it links the two unmeasured causes, and through them links the email to spend by a route that has nothing to do with the campaign. The shape is written A <- U1 -> C <- U2 -> Y and it is known as M-bias, after the letter the diagram draws.

• "It was measured before the treatment" is not a safety argument. It rules out one specific problem and says nothing about this one. Being far from the largest harm on the bench does not make it a small one.
• Composite scores are the usual carriers. A propensity-to-churn score, a health score, an engagement tier: each one is built from many inputs, which is exactly what makes it a common effect of many things.
• The unmeasured causes are what make it dangerous, which means no amount of inspecting the score itself will reveal the problem. It is a question about how the score was built.

All three harms are statements about the direction of arrows, and direction is precisely what a dataset does not contain. Two columns that move together are consistent with the first causing the second, the second causing the first, and a third thing causing both. No statistic separates those, at any sample size, which is why a correlation matrix cannot tell you whether a variable belongs in an adjustment set.

What separates them is knowledge of the business: what happens first, what the scoring model was fitted on, which team's process generates the field. That knowledge is held by operators, not by the analyst, and the diagram is simply the format that lets an operator contribute it in twenty minutes without learning any statistics.

The customer's device is in the same table, was written by the same pipeline, and looks exactly as respectable as the loyalty score. Add it to the correct adjustment set and the estimate is 6.56, with a bias of 0.33. That is the back-door set's own answer, unchanged to two decimal places. It is related to neither the email nor spend, so holding it fixed costs nothing.

That result is not a footnote. It is the reason the other three are dangerous. A covariate list contains free variables and expensive ones, and the free ones are not marked. Somebody reading six column names and asking whether they seem sensible will pass all six, because five of them are sensible-looking and one of them genuinely is harmless.

The natural hope is that a long list is a hedge: if one variable is wrong, surely the others dilute it, and the errors partly wash out. On this bench they do not. Each harm alone costs 0.97, 0.75 and 3.64. All three together give an estimate of 0.98, a bias of 5.25, which is close to their sum and nowhere near zero. They push the same direction and they stack.

On this data the cautious-sounding list is worse than no list at all. Adjusting for everything costs 5.25; adjusting for nothing costs 4.31. A team that controlled for every column it had would have been further from the truth than a team that ran the raw comparison and thought about nothing, because the post-treatment collider alone gives up 3.64.

The comparison worth quoting, though, is not one error against another. It is bias against noise. Regenerate the data under five different random draws and the correct set's residual is +0.33, +0.17, -0.38, +0.19, -0.01, averaging +0.06. It changes sign, which is what sampling noise does. Every contaminated set keeps the same sign in all five draws: loyalty about -1.2, opened about -1.0, ticket about -3.7, everything about -5.2. A wrong adjustment is bias, and it does not shrink when the sample grows. The correct one leaves noise, which does.

• A longer list is not insurance. Each additional variable is a separate bet that its role is benign, and the losing bets accumulate.
• The right set was two columns everybody already had. Season and tenure. The work was knowing which two, not collecting more.
• Both directions are available. Confounding pushed the estimate up by 4.31; the three harms pushed it down by 5.25. A team cannot even assume they are being conservative.

Everything above shrinks or inflates a number. There is a harsher version in which the number changes sign: an effect that is positive in every subgroup is negative overall, or the reverse. That is Simpson's paradox, and it is the same mechanism at a magnitude that cannot be explained away as noise.

What makes it a leadership problem rather than a curiosity is that both tables are correct arithmetic. The aggregate is right, each stratum is right, and they disagree. Choosing which one to act on is not a calculation; it is a decision about the causal structure, and the diagram is what settles it. Practitioner session 5 reproduces the reversal from real numbers and works through which table to believe and why.

None of this requires a leader to fit a model, read code or evaluate a standard error. It requires one rule, applied to every causal analysis the team ships: each variable in the adjustment set carries a written role - confounder, mediator, collider, or bystander - and one sentence saying why. Anything that cannot be labelled is labelled unknown and blocks sign-off until it is resolved.

• "It improved the fit" is not a role. Fit improves fastest for variables caused by the treatment or the outcome, which is to say for the harmful ones.
• "It was in the standard feature set" is not a role. Standard sets are assembled for prediction, where anything correlated with the outcome is welcome. Different job, opposite rules.
• "It was measured before the treatment" is not a role. The loyalty score was, and it still costs 0.97. The most expensive variable on the bench, the support ticket at 3.64, is the one measured after.
• "We adjusted for everything to be safe" is not a role. On this data it is worse than adjusting for nothing: 5.25 of bias against 4.31.

The rule is cheap to enforce and it is self-teaching. A team asked to name roles for six months stops including variables it cannot name, which removes most of the harm before anybody reaches a review.