The world that already happened
A randomized test is the gold standard because the coin flip guarantees the two groups are alike in every way, measured or not. Quasi-experiments give that up. They work with the world as it unfolded - a program rolled out to some regions first, a change that hit one market and not another - and try to reconstruct the missing counterfactual after the fact. They can be genuinely convincing. They can also be quietly wrong. The leader's job is to know which, and to hold the evidence to the right standard.
Difference-in-differences: compare the changes, not the levels 8 min live
Lumen rolls a new loyalty program out to some regions before others. You cannot compare the treated regions' sales to the untreated ones directly - they were never identical to begin with. But you can compare how much each group changed. If the untreated regions rose 5% over the period and the treated regions rose 12%, the extra 7 points is the loyalty program's effect - assuming the two groups would have moved together without it. That subtraction of two changes is the whole idea, and it is why it is called difference-in-differences.
LiveThe parallel-trends assumption, in plain words3 min▶
Difference-in-differences rests on a single "what if": would the two groups have moved together if nothing had changed? That is the parallel-trends assumption, and it is the whole ballgame.
- What it claims - absent the loyalty program, treated and comparison regions would have risen and fallen by the same amounts. Their lines would have stayed parallel.
- How you check it - look at the period before the change. If the two groups tracked together for months beforehand, the assumption is credible. If they were already diverging, it is not.
- How it breaks - if Lumen deliberately launched the program first in its fastest-growing regions, those were already pulling ahead. The "effect" would just be that head start.
Lumen launches loyalty in 4 of its 9 regions. In the year before launch, treated and untreated regions grew almost in lockstep - good news for the assumption. After launch the treated regions accelerate, opening a gap the analysts read as roughly a few points of extra conversion. Because the pre-period was clean, the CMO trusts it enough to fund a wider rollout - but commissions a proper geo experiment to confirm.
Self-studyWhy the double subtraction is clever3 min read▶
The elegance is that subtracting two changes cancels out anything shared. A nationwide promo, a seasonal dip, a supply shock - if it hit both groups equally, it lifts or drops both changes by the same amount and washes out of the difference. That is why difference-in-differences can survive a messy year that a simple before-and-after cannot.
- Before-and-after (one group) - tangles the change with every other thing that moved. Session 1's trend trap.
- Treated vs control levels (one moment) - tangles the effect with the fact that the groups were never identical.
- Difference-in-differences - subtracts both problems away, as long as the shared-trends assumption holds.
Synthetic control: build a fake Lumen that never got the campaign 7 min live
Sometimes only one region got the treatment, and no single other region is a good match for it. Synthetic control solves this by building a made-up comparison out of a weighted blend of the untreated regions - say 40% of the East, 35% of the Central, 25% of the West - tuned so the blend traces the treated region's history almost exactly before the campaign. That blend becomes your "Lumen-without-the-campaign". After the campaign, the gap between the real region and its synthetic twin is the estimated effect.
LiveThe pre-period match is the credibility test3 min▶
Synthetic control has a built-in honesty check that leaders can read without any math: how well does the synthetic twin match the real region before the campaign?
- Tight match - the synthetic line sits almost on top of the real one for the whole pre-period. That earns the right to believe the post-period gap is real.
- Loose match - if the blend could not even reproduce the past, it has no business predicting the counterfactual. Distrust the effect.
- Placebo check - good analysts also run the method pretending an untreated region was treated. If those fake "effects" are as big as the real one, the real one is probably noise.
Lumen ran its CTV burst hardest in one standout region. No single other region matched it, so the analysts built a synthetic twin from a blend of the other 8. The twin tracked the real region within a whisker for the full year before the burst, then the real region pulled clearly above it during the campaign. Because the pre-period fit was that tight, the result held up in the budget review.
Self-studyThe honest caveat: only measured confounders3 min read▶
Here is the line that separates a careful leader from an overconfident one. Both difference-in-differences and synthetic control can only adjust for the things you measured. A randomized test balances even the confounders nobody thought of, because the coin flip does not care what is measured. Observational methods cannot - an unknown factor that hit only the treated group will masquerade as effect.
- Randomized test - balances known and unknown confounders alike. Strongest evidence.
- Quasi-experiment - balances only what you modelled and the assumption you made. Good, not gold.
- The humility rule - present the effect with its assumption stated out loud, and treat it as strong evidence, not proof.
This week ◐ 30 min total
- Find a staggered rollout in your org. Something that reached some regions, teams, or segments before others. Sketch how a difference-in-differences read would work, and where you would look for the pre-period trends.
- Stress-test one parallel-trends claim. For that rollout, ask: were the treated units already pulling ahead before the change? Write one sentence on whether the assumption is safe.
- Rank three past decisions by evidence strength. Which were backed by a randomized test, which by a quasi-experiment, which by a before-and-after story? Notice how confident each one was presented versus how strong the evidence actually was.
- Optional: pick one result you were shown as fact and ask what unmeasured confounder could still explain it. Bring it to Session 6.
Three questions before you go 🎯 ◐ 90 seconds
1 · Lumen launched loyalty in some regions first. Why compare the change in each group rather than the raw sales levels?
Difference-in-differences subtracts the comparison group's change from the treated group's change. Anything shared - seasonality, a national promo, the initial level gap - washes out, leaving the treatment effect.
2 · What single thing would most convince you a synthetic-control result is credible?
If the blend cannot even reproduce the region's own history, it cannot be trusted to predict the counterfactual. A tight pre-period match is the credibility test; placebo checks reinforce it.
3 · Why is a quasi-experiment weaker than a randomized test, even with a clean pre-period?
Randomization balances known and unknown confounders alike because the coin flip does not care what was measured. Observational methods correct only for what you modelled, so they carry an assumption a randomized test does not.
What this session covers
This leader session distills the quasi-experimental toolkit the builder track builds in code. It covers ~80% of the conceptual content; the mechanics live in the sources below.