Identification before estimation

An identification failure and an estimation failure look nothing alike from the inside. A small sample announces itself: wide intervals, unstable coefficients, results that move when you refit. An unidentified estimand announces nothing at all. The model converges, the fit is respectable, the number is stable across bootstrap replicates, and it is measuring the wrong thing with great precision.

That asymmetry is why the order matters. Estimation quality is visible in the output; identification is only visible in the graph. Ten minutes with a pen answers a question that no amount of diagnostics on the fitted model will ever raise.

• Estimand - the quantity you want. Here, the total effect of the email on spend.
• Identified - the estimand can be written in terms of observable distributions, given the graph. A property of the graph plus the column list, not of the sample.
• Adjustment set Z - the variables held fixed. The criterion tests a whole set, not its members individually.
• Descendant of X - any node reachable from X by following arrows forwards. Includes X's children, their children, and so on.
• Blocked path - a path along which no association can flow, given what has been conditioned on. One blocked node is enough to block the whole path.
• Back-door path - a path from X to Y whose first arrow points into X. These are the ones that need blocking.

The first condition is a single sweep forwards from the treatment. Follow the arrows out of X, then out of everything you reach, and collect the lot. Nothing in that collection may appear in Z.

What makes this worth stating separately is how much it disposes of at once. Session 2 argued mediators out on one set of grounds and post-treatment colliders out on another; condition (i) removes both in one stroke, along with every proxy, every engineered feature built from a descendant, and every variable somebody added because it correlated beautifully with the outcome for reasons downstream of the treatment.

• At Kestrelmark the descendants of email are opened and ticket. Neither may enter Z, and neither may any feature computed from them.
• It is a purely structural test. No judgement about whether the variable is "important" or "predictive" enters into it.
• The graph does the work, so lineage matters. A feature whose definition touches a descendant is a descendant, whatever the column is called.

The second condition is the one that decides what Z has to contain rather than what it must not. List every path between X and Y, ignoring arrow directions while you walk. Keep the ones whose first arrow points into X: those are the back-door paths, and they are the routes along which association arrives without the treatment having caused anything. Every one of them must be blocked by Z.

Together the two conditions pin Z down from both sides. Condition (i) says Z may not contain consequences of X. Condition (ii) says Z must contain enough to shut every back door. What survives is the confounding structure and nothing else, which is why the criterion is a definition of "the right set" rather than a heuristic for building one.

• Paths, not variables. A single variable can appear on several paths and block some while leaving others open. The unit of the test is the path.
• Blocking is per path, not per node. One blocked node anywhere along a path blocks the whole path.
• Front-door paths are left alone. A path whose first arrow points out of X carries the effect you are measuring. Blocking one is the mediator mistake from session 2.

Walk a path node by node, looking at each intermediate node's two adjacent arrows. There are only three arrangements, and the third behaves backwards from the other two.

• Chain, i -> m -> j. Open by default. Blocked by conditioning on m.
• Fork, i <- m -> j. Open by default. Blocked by conditioning on m.
• Collider, i -> m <- j. Blocked by default. Opened by conditioning on m, or on any descendant of m.

Read the third line twice. Every intuition that says "conditioning tidies things up" is built on the first two arrangements, and the collider inverts it: doing nothing is what keeps the path shut, and touching it is what breaks the analysis. This is the same rule session 2 measured at -0.97 on the loyalty score, which precedes the email, and at -3.64 on the ticket flag, which the email helps cause: the downstream collider is by far the more expensive of the two.

Estimand: the total effect of email on spend. Candidate set: {season, tenure}.

Condition (i). Descendants of email are opened, ticket, and spend itself. Season and tenure are neither, so (i) holds.

Condition (ii). The back-door paths, meaning every path whose first arrow points into email:

• email <- season -> spend. A fork at season. Blocked, because season is in Z.
• email <- tenure -> spend. A fork at tenure. Blocked, because tenure is in Z.
• email <- u1 -> loyalty <- u2 -> spend. A collider at loyalty. Blocked by default, and it stays blocked precisely because loyalty is not in Z.
• email -> ticket <- spend. Not a back-door path at all: the first arrow points out of email. It is also a collider, so it is blocked, and adding ticket to Z would open it.

Every back-door path is blocked and no element of Z is a descendant of email, so {season, tenure} satisfies the criterion. The bench agrees: 6.56 against a true effect of 6.23, leaving +0.33. Refit on five seeds, that leftover reads +0.33, +0.17, -0.38, +0.19 and -0.01, a mean of +0.06. It changes sign, so it is noise around zero rather than bias, and more rows shrink it; every contaminated set, by contrast, keeps the same sign on all five and no amount of data moves it.

Adding device gives 6.56 again, because device sits on no path and blocks nothing, so it neither helps nor hurts. recency is the same kind of passenger as far as this check is concerned: it causes spend but not the email, so it opens no back door and {season, tenure} is still the set. Sessions 7 and 8 are where that column earns its keep.

Change one thing in the Kestrelmark graph. Suppose the disposition u1, which drives who gets the email, also drove spend directly, so that email <- u1 -> spend is a path. It is a fork, it is a back-door path, and blocking it requires conditioning on u1. There is no u1 column.

Condition (ii) now fails for every subset of the warehouse. Not for the one you tried: for all of them, because the only node that could block that path is not in the data. The effect is not identified from this data, and that conclusion is final until either the graph changes or the data does.

• More rows do not help. Sample size affects the width of an interval, not which quantity the interval surrounds.
• A better estimator does not help. Every observational estimator computes a function of the observed distribution, and the whole finding is that no such function equals the effect.
• A better fit is actively misleading. Predictive accuracy improves as the model absorbs the confounded association, so the diagnostics get healthier while the answer gets worse.

• Measure the missing variable. Often the cheapest and almost never considered, because by the time the analysis starts the column list feels fixed. Identification checked at design time is a specification for instrumentation: this field, on this event, from this date.
• Find another identification strategy. A different structure can rescue an effect the back door cannot reach. The front-door criterion is the case where a fully mediating variable is measured even though the confounder is not, and session 4 is that case in full.
• Randomise. An experiment removes the incoming arrows to the treatment by construction, which is why it needs no adjustment set. Designing, powering and analysing one is the territory of learn-experimentation-with-phoebe, and so is every observational estimator that gets reached for when randomising is impossible.
• Report the association honestly. "Emailed customers spent 10.54 more, and we cannot separate the campaign from the season and the tenure" is a defensible sentence. "The campaign drove 10.54 in incremental spend" is not, and the difference between them is one clause.

The whole check needs a graph, a list of columns and a pen. Sweep forwards from the treatment for condition (i), enumerate the back-door paths for condition (ii), and see whether anything in the warehouse blocks them all. On a graph with a dozen nodes this is a coffee break.

What that buys is a decision made before the expensive part starts. Identified, with the set in hand: proceed, and the estimation work has a target worth hitting. Not identified: stop, and spend the month on instrumenting the missing variable or on designing an experiment instead of on cross-validation.

• Do it at design time, not analysis time. The check tells you what to collect, and that is only actionable before collection starts.
• Put the verdict in writing next to the graph. Identified or not, with the set and the paths it blocks, so the next person inherits the reasoning rather than the covariate list.
• Re-run it whenever an arrow changes. The verdict is a function of the graph, so a disputed arrow that gets added can flip it.

The criterion is a theorem, and like every theorem it is conditional on its premises. It says: given this graph, this set is sufficient. It says nothing about whether the graph is right, and it cannot, because no procedure operating on the data can tell you that.

So a clean pass through both conditions is worth exactly as much as the arrows it was checked against. Get one arrow wrong and the criterion will certify a set that does not close the back door, with no warning anywhere in the output.

• Missing arrows are the dangerous kind. Session 1 made the point that a missing arrow is the stronger claim, and this is where the bill arrives: an unmeasured common cause you did not draw produces a graph on which the criterion passes.
• The response is not to abandon the method. Every alternative also rests on assumptions; the graph's virtue is that its assumptions are written down and specific enough to argue with.
• The response is to price the doubt. Ask how strong an unmeasured confounder would have to be to overturn the finding, and report that number beside the estimate. Session 9 is that calculation, and it exists because of this limit.