What the bench measures
Bias, and nothing else. Not fit, not significance, not how sensible the covariate list looks in a review. The distance between the estimate a regression produced and the effect that is actually there, which is a number no real analysis has and every learner should see once.
★ What you walk out with tonight. The measured cost of each of the three ways an adjustment set goes wrong, on one dataset, against one truth. The distinction the bench exists to make: a wrong adjustment is bias, the correct one leaves noise. Run the correct set over five seeds and its residual reads +0.33, +0.17, -0.38, +0.19, -0.01, mean +0.06, changing sign as it goes. Run any contaminated set over the same five and the sign never moves. And one result that should change how you review a model spec: the harmless variable is genuinely harmless, to two decimal places, which is exactly why the harmful ones cannot be spotted by looking at the list.
LiveWhere a known answer comes from3 min▶
Real data never comes with the answer attached, which is why this bench generates its own. The customers are produced by a structural causal model written out in the source file: season and tenure feed both the email and spend, the email feeds opening which feeds spend, and so on for eight arrows. To get the true effect, the engine runs that model twice with everything else held identical, once with the email forced on for every customer and once forced off, and differences the mean spend. That is the do-operator computed by brute force, and it gives 6.23.
The number is worth trusting for a second reason. Worked out on paper, the effect should be the direct part, which now varies by person and averages 5.161, plus the mediated part, 3.0 times the change in open rate of 1.056, which comes to 6.22. The simulation says 6.23. The engine and the algebra agree, which is the check that matters, because a simulation validated only against itself proves nothing.
Kestrelmark's actual review process was a covariate list in a pull request and two people saying it looked reasonable. Both would have approved every set below, including the one that misses by 5.25, because all of them look reasonable. The list is not readable by eye and the graph is.
Self-studyWhat is real here3 min▶
- The data. 5,000 rows from a fixed seed, identical for every learner, generated from the model in
causal-live.jswhere every coefficient is visible. - The truth. Computed by intervention, not estimated, and cross-checked against the analytic value.
- Every estimate. A real ordinary least squares fit of spend on the email plus whatever you ticked, solved by Gaussian elimination in the page. The coefficient on the email is the estimate.
- Nothing is modelled. There is no human behaviour to approximate anywhere in this widget, and no result is written down in advance.
- The one simplification. Least squares with no interaction terms, which assumes the effect is the same for everybody. Session 7 spent a whole hour on why that is false, and the bench keeps it because one lesson at a time is enough. It is worth knowing how false it is here: the model carries a
recencysegment that sets what the email is worth to each customer, and the four segments are far apart. Active customers, the sure things, gain +2.10. Customers lapsed 30 to 90 days, the persuadables, gain +29.09. Customers dormant a year, the lost causes, gain +1.04. Customers who have cancelled once, the sleeping dogs, are -8.87: a genuinely harmed group, not a rounding error. - Why
recencyis not one of the six tickable variables. It is an effect modifier, not a confounder. It changes how much the email is worth and it shifts baseline spend, but it does not cause the email, so it opens no back door and closing it buys nothing. The back-door set is still season and tenure, and every graph from sessions 1 to 3 stands unchanged.
Six adjustment sets, one truth
10 min liveEvery row below is computed by the bench on this page against a true effect of 6.23. Read the sign of the bias as carefully as the size: the direction tells you which mechanism is at work, and whether the sign holds still under a new seed tells you whether you are looking at bias or at noise.
| Adjustment set | Estimate | Bias | What is happening |
|---|---|---|---|
| Nothing | 10.54 | +4.31 | Two back doors wide open: season and tenure |
| season, tenure | 6.56 | +0.33 | The back-door set. What is left is sampling noise, not error |
| + loyalty score | 5.26 | -0.97 | A pre-treatment collider. Conditioning opens a path that was closed |
| + opened | 5.48 | -0.75 | The mediator. Now measuring the mean direct effect, 5.16, not the total |
| + ticket | 2.59 | -3.64 | A post-treatment collider, and by far the most expensive single variable here |
| + device | 6.56 | +0.33 | A bystander. No change at all, to two decimals |
| Everything in the table | 0.98 | -5.25 | All three harms at once, pushing the same way |
The three readings that matter. First, confounding inflated the estimate by two thirds of the true effect again, and no amount of extra data would have touched it. Second, adjusting for the mediator returns 5.48, which is close to the mean direct effect of 5.16: not a mistake so much as an answer to a different question, and the spec has to say which question. Third, and least expected, the three harmful adjustments all push downward and compound, reaching 5.25 rather than partly cancelling.
Noise versus bias, which is the real lesson. The correct set's +0.33 is not a small error, it is the absence of one. Re-run the whole bench on five different seeds and the back-door set gives +0.33, +0.17, -0.38, +0.19, -0.01, a mean of +0.06, and it changes sign. That is what a residual looks like when nothing is systematically wrong. Every contaminated set keeps the same sign in all five seeds and lands in the same place each time: loyalty about -1.2, opened about -1.0, ticket about -3.7, everything about -5.2. More rows shrink the first kind and leave the second exactly where it is.
The size order is not the one a review expects. The pre-treatment collider, the one everybody argues about because its timing makes it look defensible, is not the dearest of the three: it costs 0.97, while the mediator costs 0.75. The dearest by a wide margin is the post-treatment collider at 3.64, and ticket is exactly the kind of column that gets added without argument: recent, well populated, and it improves the fit.
Which is why the full list here loses to the empty one. Adjust for everything and the bias is 5.25; adjust for nothing at all and it is 4.31. An analyst who controlled for every column Kestrelmark had would have been further from the truth than one who ran the raw difference and went home.
Score the sets yourself
★ 25 min · everyone buildsThe bench is live below. Each variable is labelled with its role in the graph, which is information you would not have in real life; the point of tonight is to feel what each role does before you have to guess.
Start with nothing ticked. 10.54 against a truth of 6.23. This is the number a dashboard produces and the number the growth team took to the board. Every path from email to spend is open, including the two that are not the email's doing.
Tick season, then tenure. Watch the estimate fall to 6.56 as each back door closes. The residual of 0.33 is worth pausing on, because it is the only number tonight that is not an error: change the seed and it reads +0.17, then -0.38, then +0.19, then -0.01, mean +0.06, wandering either side of zero. The correct answer was available all along from two columns everybody had.
Add the loyalty score. It was computed before the email went out, which is the reason people call a variable safe, and the estimate drops to 5.26. Loyalty is a common effect of two unobserved things, one of which feeds the email and one of which feeds spend. Conditioning on it connects them. The cost, 0.97, is the middle of the three harms, and it survives every seed at about -1.2 while the correct set's residual keeps flipping.
Untick loyalty, add opened. 5.48. Compare that with the mean direct effect of 5.16 in the model. You have not made an error; you have answered "what does the email do for people whose opening it did not change", which is a real question and not the one on the spec.
Untick opened, add ticket. 2.59, a bias of 3.64, and this is the moment of the session. Same mechanism as loyalty, but this collider sits after the treatment, and it is far and away the most expensive of the three, well ahead of the 0.97 that the pre-treatment collider costs. Ticket is the variable most likely to be in a real model, because it is recent, well populated and correlates beautifully with everything.
Untick ticket, add device. 6.56, unchanged. A variable related to nothing costs nothing. Sit with that for a moment: the list of six looks homogeneous, and one member is free while three are expensive and one of those three is expensive on its own.
Press "everything in the table". 0.98, a bias of 5.25. Three harms in the same direction, compounding. If your instinct was that they might partly cancel, that instinct is now measured and wrong. Note where that leaves the full list against the empty one: 5.25 against 4.31, so on this data the analyst who controlled for everything did worse than the analyst who controlled for nothing.
Write the covariate list you would defend in a review, and next to each variable write the role you believe it plays. Any variable whose role you cannot name is a variable you are including by habit.
What the bench leaves out, deliberately. The regression fits one number for everybody, and the generating model is not that simple at all: the email is worth +2.10 to an active customer, +29.09 to one lapsed 30 to 90 days, +1.04 to one dormant a year, and -8.87 to one who has already cancelled once. Every set you tick tonight is averaging over those four. Session 7 recovered them and session 8 built the targeting that follows, but stacking heterogeneity onto this bench would blur the one thing it is built to show. Add it to your own version second, not first.
Closing the track
45 min- Take the covariate list from a model your team already ships and write the graph role beside every variable: confounder, mediator, collider, bystander, or unknown. Count the unknowns.
- Find one variable in that list that is caused by the treatment. There is usually one, and it is usually there because it improved the fit.
- Run the session 1 graph and the session 3 identification check on that model and write down whether the estimand is identified from the data you have.
- Put the graph, the estimand and the adjustment set in the repository beside the query, and make the next review a review of the graph rather than the list.
Three questions
Adjusting for the loyalty score, which was measured before the email was sent, moves the estimate from 6.56 to 5.26. Why is "it happened before the treatment" not a safety argument?
Timing says nothing about the shape. What matters is whether the variable is a common effect, and a pre-treatment collider does the same damage as a post-treatment one.
Adjusting for whether the customer opened the email gives 5.48, against a true total effect of 6.23 and a mean direct effect of 5.16. What has the analyst done?
The open is the mechanism, not a nuisance. Conditioning on a mediator is only an error if the spec asked for the total effect, which is why the estimand goes in writing before the covariate list.
The three harmful adjustments cost 0.97, 0.75 and 3.64 alone. Together they cost 5.25, against 4.31 for adjusting for nothing at all. What does that tell you?
5.25 is close to the sum and nowhere near zero, and it is larger than the 4.31 you get by controlling for nothing. Adding more controls in the hope that errors wash out makes the estimate worse, not more robust.