Commissioning and reading a causal claim

A causal analysis has a fixed order of decisions, and the expensive ones come first. What is being intervened on. What quantity is wanted. Whether the data can deliver it. What must not be held fixed. Who the answer is about. Only after all five is there a choice of technique, and the technique is the part that gets all the attention in a review because it is the part that looks technical.

The consequence is uncomfortable and useful. A brilliant method applied to an unstated estimand produces a precise answer to an unknown question, and there is no way to tell from the output. A crude method applied to a well-specified estimand with a written identification argument produces something a reviewer can actually check. The brief is where a leader has full authority and needs no statistics, and it is also where the outcome is mostly decided.

• Estimand - the quantity you want, stated before any recipe exists. "The total effect of sending the email on 90-day spend, for customers who lapsed in the last quarter." A sentence, in the language of the business, that could be written by somebody who has never fitted a model.
• Identification - whether that quantity can be recovered from the data available, given what is believed about how the world works. A yes-or-no question that precedes estimation and is answered on a diagram rather than in a table.
• Adjustment set - the specific variables held fixed, and nothing else. Not the standard feature set, not everything that improved the fit, but a named list with a named reason per line.
• Sensitivity - how far the conclusion survives being wrong about the assumptions. Concretely: how strong an unmeasured confounder would need to be before the finding reverses.

Those four words are the whole grammar of the brief. Three of them describe decisions made before any data is touched, and the fourth describes the honesty of what comes back.

The first line of the brief is a question containing a verb somebody could act on. "What would 90-day spend have been if we had sent the email to every lapsed customer, compared with sending it to none of them?" That sentence names the intervention, the outcome, the window and the comparison, and it can only be answered on rung 2.

Beside it, in brackets, goes the rung. Rung 1 is a question about what is observed together: who spends more, which segments correlate with churn, what the emailed group looks like. Rung 2 is a question about doing something. Rung 3 is a question about an individual world that did not happen: would this customer have churned anyway. Naming the rung takes four seconds and it prevents the single commonest failure, which is a rung-2 question answered with a rung-1 comparison and reported with a causal verb attached.

• If the sentence has no intervention, the brief is not a causal brief. That is fine, and it should be relabelled rather than upgraded quietly later.
• The comparison matters as much as the intervention. Against nobody, against the current policy, against a cheaper alternative. Three different numbers, one ambiguous request.
• The window belongs in the question. A 7-day and a 90-day effect of the same email are different quantities and both are legitimate.

The second line says which effect is wanted, and in practice the choice that matters is between the total effect and the direct effect. The total effect is everything the intervention does, by every route including the ones you would rather it did not use. The direct effect is what survives when the main mechanism is held fixed.

This is not a philosophical distinction; it is a number. On Kestrelmark's data the total effect averages 6.23. Hold the open event fixed, which is the mechanism through which much of the email works, and the estimate becomes 5.48, close to the mean direct effect of 5.16 written into the model that generated the data. Nothing in the output announces which of the two you received. Both are a clean coefficient with a sensible sign, and adding one variable to a covariate list is enough to switch between them silently.

• The total effect answers "should we do this". Nearly always the commercial question, and the default for a rollout decision.
• The direct effect answers "how much survives without the mechanism". A real question for a subject-line project or a channel redesign, and a different one.
• An unwritten estimand is chosen by whoever writes the query. Not carelessly, but privately, and the choice never appears in the readout.
• An estimand also has to say who it is about. Kestrelmark's average of 6.23 is made of four very different numbers: +29.09 for the 20.4 percent who lapsed recently, +2.10 for the 22.4 percent still active, +1.04 for the 49.3 percent long dormant, and -8.87 for the 7.9 percent the email drives away. At a fully loaded 7.00 per send the blanket campaign is worth -3,848 and sending only to the lapsed segment is worth +22,578. Session 5 is where that is worked through; the brief's job is to require it, by naming the population and asking for the answer by segment as well as overall.

The third part is a short paragraph in plain English saying which contaminating routes exist between the intervention and the outcome, and which measured variables close each one. For Kestrelmark it reads roughly: season causes both the send and the spend, tenure causes both the send and the spend, and holding those two fixed closes both routes, which is why the estimate moves from 10.54 to 6.56 against a true average of 6.23.

The paragraph has a second half, and the second half is where the value is: what to do if one route runs through something nobody measures. That case is not rare, it is the normal case outside a controlled setting, and there are only four honest responses. Measure the thing. Find a different route to the answer. Report the effect with an explicit statement of what would have to be true. Or conclude the effect is not identified from this data and say so early.

• It is written before the data is pulled. Written afterwards, it becomes a justification for whatever was done, which is a different document with the same words in it.
• It is legible to an operator. Somebody who runs the process should be able to read it and say "that route does not exist" or "you have missed one". That review is worth more than any technical check.
• It names the variables by their warehouse names. "We control for seasonality" is not an identification paragraph. Which column, from which table, at which grain.

The fourth part is a list of variables with a named role and one sentence each. Confounder, mediator, collider, bystander. Anything that cannot be labelled is labelled unknown, and an unknown blocks sign-off until somebody resolves it. This is the shortest part of the brief and it carries the most weight, because the costs are large and invisible: on Kestrelmark's bench the correct two-variable set leaves +0.33, and holding every available variable fixed lands at -5.25, further from the truth than holding nothing fixed at all, which leaves +4.31. The harmful controls all push the same way and stack.

• "It improved the fit" is not a role. Fit improves fastest for variables caused by the treatment or by the outcome, which is to say for the harmful ones.
• "It was in the standard feature set" is not a role. Standard sets are assembled for prediction, where anything correlated with the outcome is welcome. Different job, opposite rules.
• "It was measured before the treatment" is not a role. Timing rules out one specific problem and is silent about the others.
• An empty adjustment set is a role list too. If nothing needs holding fixed, write down why, because that is a strong claim and it should be visible.

The rule is self-teaching. A team asked to write roles for six months stops proposing variables it cannot name, and most of the harm disappears before anything reaches a review.

The last part asks the analysis to state its own fragility. Every observational causal claim rests on the assumption that no important common cause was missed, and that assumption is never verifiable from the data. What can be done instead is to ask how strong a missing cause would have to be before the finding stops holding.

The answer is a threshold, and thresholds are readable by non-specialists. If a conclusion survives any plausible unmeasured factor, that is a strong result and should be said loudly. If it reverses under a factor half as strong as tenure, the finding is fragile and the brief should have said so before the recommendation was written. Either way the number belongs in the readout rather than in a footnote nobody reaches.

• Ask for it in the brief, not in the review. Requested afterwards it reads as scepticism about a specific result; requested upfront it is simply part of the deliverable.
• Compare the threshold with something known. "Would need to be twice as strong as tenure" is a sentence a leader can weigh. A bare coefficient is not.
• Practitioner session 9 works the technique properly, including what to do when the threshold is embarrassingly low.

Each of these is a property of the written claim and can be spotted without opening anything. None of them is an accusation of incompetence, and saying so out loud when you send it back matters, because the failure is nearly always in the brief rather than in the work.

• A causal verb attached to a rung-1 comparison. "The campaign drove 10.54 units of incremental spend" over a plain difference between the emailed and the not-emailed. The word drove, the word incremental and the word impact all promise rung 2. The sentence to say: "which contaminating routes were closed, and by which variables?"
• A covariate list containing anything measured after the treatment. One sweep, one question per variable: could the intervention have changed this? Anything with a yes is a mediator or a collider, and both move the answer. The sentence to say: "which of these were recorded after the send, and what role does each one play?"
• A claim with no statement of what would make it wrong. If nothing could falsify it, nothing was tested. The sentence to say: "how strong would an unmeasured common cause have to be to overturn this?"

Three sentences. None requires knowing a method, and all three are answerable in a day by whoever did the work, which is the test of whether they were fair to ask.

The reverse failure is real and less discussed. Three deliverables get pushed back on regularly and should be signed off without argument, because refusing them teaches a team to overclaim.

• An honest association with no causal verb. "Customers who received the email spent 10.54 more" is a true, useful, correctly hedged sentence. Sending it back for being "not actionable enough" is a request for the causal verb without the causal work, and the team will supply it.
• A wide interval. Width is information about how much the data supports. A wide honest range from a correct adjustment set is worth more than a narrow one from a set that includes three harmful controls, and the narrow one will look better in every meeting.
• "Not identified from this data." A conclusion that the question cannot be answered with what exists, arrived at in week one, is a genuine deliverable. It stops a quarter of work that would have produced a confident number of unknown sign, and it converts into a concrete data-collection request or an experiment.

One habit makes all of the above durable rather than a good quarter. The diagram, the estimand and the adjustment set live in version control, in the same directory as the query, in plain text. Not in a slide, not in a document nobody can find, not in the analyst's head.

The reason is what it does to the next review. When the assumptions are text files beside the code, a change to the covariate list appears as a diff, the person making the change has to write the role to get it merged, and a reviewer six months later can see who added which arrow and when. The review stops being an argument about whether a number looks right and becomes an inspection of assumptions, which is the only review that can actually catch anything.

• Plain text, one line per arrow. A picture in a deck cannot be diffed and will describe last quarter's beliefs within a month.
• The estimand as a comment at the top of the query. One sentence. It survives staff changes, which slides do not.
• The role list as a file the pull request touches. That is the entire enforcement mechanism, and it costs one line per variable.

The brief stops before the method, and the omission is intentional. A leader who specifies the technique has spent their authority on the one decision the analyst is better qualified to make, and has usually done it by naming the technique they heard about most recently.

• The estimation machinery belongs to a different course. Propensity scores, inverse probability weighting, doubly robust estimation, difference-in-differences, synthetic control and the DoWhy workflow are all taught properly in learn-experimentation-with-phoebe. Naming one in a brief signals confidence rather than rigour, and it can quietly override the estimand written in part 2.
• Running an experiment is also there. When the identification paragraph concludes that a back door cannot be closed, the honest next step is often a randomised comparison, and the design of that comparison is that course's territory.
• Statistical inference is elsewhere again. Standard errors, p-values and power belong to learn-statistics-with-phoebe. The brief asks for an interval and does not specify how it is constructed.

What the brief keeps is everything that cannot be delegated: the question, the quantity, the identification argument, the roles, and the demand to state what would make the answer wrong.