Draw the DAG before touching the data

A dataset cannot tell you which of two variables causes the other. The same table is consistent with the email raising spend, with high spenders being selected for the email, and with a third thing driving both. Choosing between those three is not a statistical operation; it is an assumption, and the graph is where the assumption gets written down where a colleague can attack it.

That is the whole argument for drawing it first. Once the graph exists, a set of purely mechanical questions become answerable: which paths carry real effect, which carry contamination, what has to be held fixed, and whether the data you have is even capable of answering the question. Without it, adjustment is a matter of taste, and session 10 measures what taste costs.

• DAG - a directed acyclic graph. Directed because arrows point from cause to effect, acyclic because nothing causes itself through a loop.
• Path - any chain of arrows between two variables, regardless of which way they point. Some paths carry causation, some carry only association.
• Back-door path - a path from treatment to outcome that starts with an arrow pointing INTO the treatment. These are the contaminating ones, and closing them is the job. Session 3 states the criterion formally.
• Confounder - a common cause of treatment and outcome. Opens a back-door path; adjust for it.
• Collider - a common effect of two variables. Conditioning on one creates association where there was none. Session 2.
• Mediator - sits on the causal path from treatment to outcome and carries part of the real effect. Adjusting for one answers a different question. Session 2.
• Identification - whether the effect can be computed from the data you have, given the graph. Separate from, and prior to, estimation. Session 3.
• Estimand, estimator, estimate - the quantity you want, the recipe, and the number you got. Confusing them is how teams answer a question nobody asked.

The test for whether a graph is honest is whether every arrow survives being said out loud to somebody who knows the business. Kestrelmark's:

• Season into email. The winter campaign calendar sends more email. Nobody disputes this.
• Season into spend. People buy more in winter regardless of email. Also undisputed, and together with the arrow above this is what makes season a confounder.
• Tenure into email and into spend. Long-tenured accounts are on more lists and buy more anyway.
• Email into opened into spend. The email works partly because people open it. This is the mechanism, not a nuisance.
• Email into spend, directly. Even unopened, the sender name in an inbox is a reminder. Arguable, and worth arguing about.
• Email and spend both into ticket. Getting the email prompts questions; spending more prompts questions. Two arrows into one box.

Six sentences, five minutes, and the shape of the analysis is now fixed. Notice that not one of them required looking at the data.

Kestrelmark's graph has no arrow from device into spend. That says mobile users do not spend differently, which is a real claim and would be worth ten minutes of checking. It also has no arrow from ticket back into spend, which says that raising a ticket does not itself change what somebody buys. Plausible for this business, false for many others.

This is the honest weakness of the whole method and it is better stated than hidden: the conclusions depend on arrows you chose not to draw, and no dataset will tell you that you were wrong. What the graph buys is that the assumption is visible and specific, so a colleague can point at one missing arrow and say "that one, I think that one exists" - which is a conversation you can have, unlike a disagreement about a number.

Kestrelmark's u1 and u2 are drawn as dashed circles because nobody measured them. u1 is whatever disposition makes somebody both a target for the campaign and a certain kind of loyal; u2 is whatever makes them both loyal and a spender. Leaving them off the graph because they are not columns in the warehouse would be the single most common mistake in applied work.

• They change what is identifiable. A back-door path running through an unmeasured variable cannot be closed by adjustment, which is a fact about your data you want to know before the analysis, not after.
• They are why the loyalty score is dangerous. Loyalty is a common effect of u1 and u2. Session 2 shows what conditioning on it does; session 10 measures it at -0.97.
• They are the target of sensitivity analysis. If you cannot measure it, you can still ask how strong it would have to be to overturn your finding. Session 9.

Comparing the emailed group to the not-emailed group measures every open path at once. Two of those paths are the effect you want and two are the season and the tenure leaking in. The arithmetic in session 10 puts the leak at 4.31 units, which is two thirds of the true effect again on top.

Nothing about this is fixed by more data. A million rows gives you 10.54 with tighter error bars, which is a more confident wrong answer. It is fixed by holding season and tenure fixed, and knowing that those two are the right things to hold fixed comes from the graph rather than from the table.

Kestrelmark's email works two ways: some people open it and buy, some people see the sender and buy without opening. The total effect, 6.23, is both together. The direct effect, a mean of 5.16, is only the second.

• The total effect answers "should we send the email". That is nearly always the business question, and it is the estimand for the rest of this track.
• The direct effect answers "how much of this survives if nobody opens it". A real question for a subject-line project, and a different one.
• Adjusting for opened silently switches you from the first to the second. Session 10 measures the switch at 5.48, roughly the mean direct effect of 5.16, and session 2 explains the mechanism.

Write the estimand down in the graph's caption before adjusting anything. It takes one sentence and it prevents the most common way a correct analysis answers the wrong question.