Identifiable or not

The expensive pattern is not a wrong answer. It is three months of competent work that ends with a number nobody can defend, discovered at the review, by which point the roadmap has been built on it. The team was skilled, the code was clean, the sample was large, the pipeline was tested. The question was simply not answerable from the data, and nothing in the workflow asked.

What makes it expensive is that the failure is silent. An analysis that cannot identify its effect still produces a coefficient, a confidence interval and a chart. It does not warn. Every diagnostic a team routinely runs - fit, stability, out-of-sample performance, cross-validation - is a diagnostic on the estimate, and none of them can see the problem, because the problem is not in the data.

• Estimand - the quantity you want. "The total effect of sending the re-engagement email on 90-day spend, across all subscribers." A sentence, written before anything else, with an intervention in it.
• Identification - whether that quantity can be computed from the data available, given how the world works. A yes or a no, decided by structure and by which columns exist, not by sample size and not by technique.
• Estimation - the recipe that turns the data into a number, once identification has said yes. This is where sample size, precision and method live, and it is the only one of the three most teams discuss.

The order is fixed and it is not negotiable: estimand, then identification, then estimation. A team that starts at the third is choosing a route before checking there is a destination.

Estimation asks how precisely. Given that the quantity can be recovered, how tight is the answer, and would more data help? This is the familiar territory: sample size, variance, error bars. More data always helps here, which is why more data feels like the universal remedy.

Identification asks whether at all. Given how customers came to be treated, is the quantity recoverable from these columns by any procedure? This is a yes or a no. When it is no, more data makes the wrong answer more precise, and precision on a wrong answer is worse than noise, because noise looks like uncertainty and precision looks like knowledge.

• A failure of estimation looks like a wide interval. Visible, honest, and fixable by collecting more rows.
• A failure of identification looks like a clean result. Invisible, and not fixable by any amount of anything you are currently collecting.
• Every routine diagnostic tests the first. Fit, stability, holdout performance. None of them can see the second, because the second is a claim about the world rather than about the table.

There is a precise test for whether a set of variables is the right thing to hold fixed. A set Z satisfies the back-door criterion relative to the treatment X and the outcome Y if (i) no member of Z is a descendant of X, and (ii) Z blocks every path between X and Y that contains an arrow INTO X.

In plain English, two rules. Nothing you hold fixed may be a consequence of the treatment - that is condition (i), and it is what ruled out the mediator and the post-treatment collider in session 3. Every route from treatment to outcome that runs backwards, through a shared cause, must be shut by something in your set - that is condition (ii), and it is what makes Z the right list of confounders rather than a plausible one.

The identification question is then simply this: does a set satisfying both conditions exist, and are all of its members columns you actually have? A set can satisfy the criterion perfectly and still be useless because one of its members was never recorded, and that distinction is the whole of tonight.

Two things make a Kestrelmark customer more likely to get the re-engagement email and also more likely to spend. Winter, because the campaign calendar sends more in winter and people buy more in winter. And tenure, because long-standing accounts are on more lists and buy more anyway. Two back-door routes, and both of them run through a variable the warehouse already holds.

So the set {season, tenure} satisfies both conditions: neither is caused by the email, and between them they shut both backwards routes. The effect is identified. Holding those two fixed produces 6.56 against a true average effect of 6.23, a residual of +0.33. The naive comparison, with both routes left open, gives 10.54.

• The gap between 10.54 and 6.56 is 4.31, two thirds of the true average again, and it came from two columns nobody thought to use.
• The 0.33 left over is noise, not accuracy. Run the same correct adjustment on five different draws of the same 5,000 customers and the residual reads +0.33, +0.17, -0.38, +0.19, -0.01, a mean of +0.06. It changes sign, which is what a quantity wobbling around zero does. Every contaminated adjustment set keeps the same sign in all five. That is the difference identification buys: an identified question leaves noise, which more data shrinks; an unidentified one leaves bias, which more data will not touch.
• Nothing was collected to fix it. The right answer had been available all along; the missing piece was knowing which two columns mattered, which came from the diagram.
• The verdict was available in week one, before a single query, from a whiteboard conversation with somebody who knew the campaign calendar.

Now change one detail. Suppose Kestrelmark's targeting had not been calendar-driven but had run on an account manager's judgement about which customers seemed at risk, and suppose that judgement was never written down. There is now a back-door route from the email to spend running through something nobody records: the manager saw a signal, that signal drove the send, and the same signal drove spend independently.

Every condition (i) rule still passes. The team can still assemble a sensible-looking adjustment set. But condition (ii) fails permanently, because the only variable that would close that route does not exist as a column. The effect is not identified from this data, and that verdict is unchanged by sample size, by choice of method, by anything at all.

Often the cheapest real fix, and almost always the one nobody proposes, because it is a data collection project rather than an analysis. If the account manager's risk judgement is the open back door, then logging that judgement - a dropdown, a required field, a reason code on the save offer - turns an unanswerable question into an ordinary one.

• The cost is calendar time, not analyst time. Weeks to a quarter before there is enough history to analyse, which is precisely why finding out in week one rather than month three matters so much.
• It is permanent. Every future question that runs through that same confounder becomes answerable, so the value is not confined to the analysis that triggered it.
• Proxies are a partial fix at best. A variable correlated with the missing one closes the path partially, which reduces the bias by an unknown amount. Better than nothing, and not the same as identified.

Closing the back doors is not the only route to a causal answer. When a confounder is unmeasured but the mechanism through which the treatment works is fully observed and has no back door of its own, the front-door argument can recover the effect without ever measuring the confounder. Practitioner session 4 works through when that applies and what it demands.

The thing a leader needs to know is the trade being offered. This route does not remove assumptions; it swaps one set for another. Instead of assuming you measured every confounder, you assume the mechanism is complete and unconfounded. That may be easier to defend or much harder, depending on the business, and the honest way to present it is as a different bet rather than a clever escape.

Randomising who gets the treatment severs every arrow into it, which closes every back-door path at once, including the ones through variables nobody has ever recorded. That is the whole reason experiments occupy the position they do: identification is bought by design rather than argued from a diagram.

The cost is real and worth naming: eligible traffic, calendar time, a group deliberately held back, and the organisational willingness to not treat people you believe would benefit. Where the treatment cannot be withheld or the population is too small, this route closes. How to design, size, run and read one is the territory of learn-experimentation-with-phoebe, which covers it in full.

This is a legitimate deliverable and it is treated as a failure far too often. "Customers who received the email spent 10.54 more" is true, reproducible, useful for forecasting and for segmentation, and it makes no claim the data cannot support. What is forbidden is the second sentence, where the same number becomes "the email drove 10.54 units of incremental spend".

• The discipline is verbs. Drove, caused, generated, lifted, incremental, uplift, impact. Ban them from the deliverable and the honesty enforces itself, because the sentences that remain are the ones the data supports.
• Say what would change the verdict. "This becomes a causal claim if we log the targeting reason" turns a limitation into a roadmap item somebody can fund.
• It ships this week. Alone among the four, this option has no calendar cost, which is what makes it the right answer when the decision cannot wait.

There is one more move, and it is honest as long as it is not oversold. Rather than closing the open path, you ask how strong the unmeasured cause would have to be to overturn your conclusion. If the answer is "stronger than any variable we do measure, by a factor of three", the finding is fragile in principle and robust in practice. If the answer is "about as strong as tenure", it is not a finding.

What it does not do is make the effect identified. It converts an unqualified claim into a conditional one with a stated breaking point, which is a genuine improvement in a brief and not a substitute for any of the four options above. Practitioner session 9 works through how the number is produced and how to read it.

The identification check costs about twenty minutes and needs no data. The failure it prevents costs a quarter: a team models for three months, a reviewer asks how people ended up in the treated group, and the honest answer retroactively invalidates everything built on the result, including whatever the roadmap already committed to.

The asymmetry is what makes this a leadership habit rather than an analyst's technique. The check is free and the analyst cannot always run it alone, because the crucial fact - how customers actually came to be treated - lives with operations, sales or the campaign team. Whoever commissions the work is the only person who can reliably get those people into the same twenty minutes.

• Twenty minutes against a quarter. The worst case of running the check is that it confirms a yes and the team proceeds with more confidence.
• The verdict belongs in the brief, above the method. If it is written down at the start, nobody can be surprised at the review.
• A no found in week one is a good week. It is the cheapest useful thing an analysis can produce, and it should be received that way.