Sensitivity analysis

"There could be an unmeasured confounder" is a statement about the space of possible worlds, and that space is infinite. No dataset refutes it, because the variable is by definition absent from the dataset. No sample size touches it, because it is a bias rather than noise. Nobody can prove it wrong, which means nobody can be argued out of it, which means the sentence has no work to do beyond stopping the meeting.

It is also useless to whoever says it. The objector cannot say how much it would matter, so they cannot distinguish a finding that would collapse under a faint breeze from one that would need a hurricane. Both get the same objection and the same shrug. A room that cannot tell those two apart will end up either believing everything or believing nothing, and both outcomes are expensive.

• After identification, not instead of it. Session 3 asks whether a valid adjustment set exists in the data. Sensitivity analysis assumes you already believe one does and asks what an unnamed extra cause would have to look like to overturn the result.
• Not a repair. Nothing here removes bias. It reports how much bias would be needed, which is a different and more honest thing to hand a reader.
• Not an estimator. Correcting an observational comparison, by reweighting or otherwise, belongs to learn-experimentation-with-phoebe. What is built tonight sits on top of whatever estimate you already have.
• The published families have names. You will meet them as Rosenbaum bounds, E-values, and covariate-benchmarked partial-association bounds among others. Each assumes a different shape for the unmeasured cause, and that assumption is the first thing to read in any of them.

Both sides of the argument are wrong in the same way, which is worth saying out loud because it defuses the room. The analyst who says "we controlled for everything relevant" is claiming to have enumerated the causes of human behaviour. The sceptic who says "there could be something else" is claiming that possibility alone is a reason to disbelieve. Neither statement can be checked, and the two of them can be repeated at each other indefinitely.

• Always true. There is always something unmeasured. Spend at Kestrelmark is caused by things nobody has a column for, and always will be.
• Always unfalsifiable. A variable you did not measure produces no evidence about itself, in either direction.
• Therefore useless as stated. A sentence that applies identically to every observational finding ever produced cannot help anyone rank two of them.

Notice what the objection is not. It is not a general complaint about causal claims from data: a randomised assignment answers it by design, which is the reason randomisation is worth its cost. It is specifically about what happens when the assignment was made by the world instead.

Do not ask whether an unmeasured confounder could exist. Ask this instead:

Three things change at once. The question now has a numerical answer, so it can be computed rather than asserted. It requires the confounder to do two jobs at once, associating with the treatment and with the outcome, which is a much harder bar than doing either alone and is where most speculative confounders fail. And the last clause, is anything that strong plausible, hands the judgement to whoever knows the domain, which is usually not the analyst and is exactly the person whose objection started this.

Because it converts an argument about possibility into a comparison against variables you HAVE measured. Possibility has no scale and therefore no stopping rule. Measured covariates have both: they are real numbers, computed from the same data, on the same footing, and everyone in the room already accepts them.

That comparison is what a domain expert can actually rule on. Nobody has a well-formed opinion about whether an association of 0.31 is large. Plenty of people have a strong opinion about whether there is a driver of subscription spend more than one and a half times as important as how long somebody has been a customer, and if they say no, the finding has survived something. If they say yes and can name it, the finding has failed usefully, because now you know what to go and measure.

Take the covariates already in the adjustment set and measure, for each, how strongly it is associated with the treatment and with the outcome. Those pairs are the ruler. Then express the tipping point as a multiple of the largest of them. If an unmeasured cause would need to be three times as strongly associated with spend as season is, and season is the strongest thing measured, the finding is robust: something bigger than the biggest known driver, hiding in plain sight, is a specific and unlikely object. If it only needs to be half as strong as tenure, it is not robust, because things that size are common and one of them is probably missing.

• Compare per standard deviation, not per raw unit. At Kestrelmark, season enters spend with a coefficient of 6.0 and tenure with 5.0, so season looks stronger. Season is binary and half the customers are in winter, so one standard deviation of it moves spend by 3.0 against tenure's 5.0. Per standard deviation, tenure is the stronger benchmark, and picking the wrong one here inflates the robustness claim by more than half.
• Benchmark on both associations at once. Tenure's pair at Kestrelmark is 0.7 into the email and 5.0 into spend. A candidate that matches tenure on one and is trivial on the other is not a tenure-strength confounder.
• Benchmark against what is IN the model. The point of the ruler is that the reader already believes it. A covariate you excluded proves nothing about a covariate you never had.

State the tipping point alongside the estimate, in the same paragraph, in one sentence, so a reader can judge the claim without rerunning anything or opening an appendix. It costs a line and it removes the single most common reason an observational finding gets waved away.

Read what the sentence does not claim. It does not say no such cause exists. It does not say the estimate is correct. It says precisely how large the missing thing would have to be, in a unit the reader already has intuition for, and then it stops and lets them decide. That restraint is why it survives review.

• It assumes a shape. Every sensitivity method fixes a form for the unmeasured cause: one variable rather than several, additive rather than interacting, the same strength for everybody. The tipping point is correct for that shape and silent about others. Two methods disagreeing on the same data usually means they assumed different shapes, not that one is broken.
• It cannot rule anything out. A tipping point of 1.55 tenures does not mean no such confounder exists. It means one would have to be that big, and somebody who knows the domain still has to say whether that is plausible.
• It says nothing about the arrows you drew wrong. This is the one that costs people their conclusions. Sensitivity analysis answers a question about missing confounding. It has no opinion about a variable that should never have been in the model at all.
• A collider you conditioned on is not a confounding problem. Conditioning on a common effect creates association rather than failing to remove it, and no sensitivity analysis will find it. Kestrelmark's loyalty score costs 1.30 units of bias while every sweep run on top of it reports a comfortable number. That is why sessions 2 and 3 come first: sensitivity analysis is the last check, never the first, and it cannot rescue a graph that was wrong.

Leader session 4 makes the case that "this effect is not identified from the data we have" is a legitimate finding rather than a failure to deliver, and a tipping point is what makes that finding actionable instead of merely discouraging. The pairing is the deliverable: here is the best estimate available, here is the honest statement that a back-door path runs through something unmeasured, and here is how strong that unmeasured thing would have to be for the decision to flip.

• Sometimes the tipping point is enormous and the decision can be made anyway. A finding that would need a confounder five times the size of anything known is good enough for a reversible, low-cost decision, and saying so is more useful than another month of modelling.
• Sometimes it is tiny and the honest output is a measurement request. Name the variable you would need, say what it would cost to collect, and hand back a design rather than an estimate.
• Either way the decision-maker gets a size. That is the difference between a caveat and information, and it is the reason this sentence belongs in the summary rather than the appendix.