Geo experiments and synthetic control

Lumen wants to test a connected-TV burst - a two-month brand campaign running on CTV in a handful of US metros. You cannot randomize which shopper "sees TV"; the delivery is inherently geographic, and post-cookie you cannot even track it cleanly at the user level. So you randomize or select regions instead: treat a subset, hold the rest out, and reconstruct what the treated regions would have done without the burst. Session 8 teaches the two credible ways to build that counterfactual - Google's CausalImpact and the synthetic control method (SCM) - and, just as important, how to earn trust in the estimate with a good pre-period fit and placebo inference.

A geo experiment makes the region the unit of randomization. You split markets (DMAs, states, metros) into treated and control, run the intervention in the treated markets only, and compare aggregate outcomes. This is the right tool when treatment cannot be delivered per user - broadcast and CTV media, pricing, a policy change - or when privacy loss (Safari/Firefox/ATT) has gutted user-level tracking. The cost is fewer units: you might have 9 regions, not 64,000 users, so the analysis leans hard on modeling the counterfactual well.

The synthetic control method (Abadie & Gardeazabal 2003; Abadie, Diamond & Hainmueller 2010) builds a synthetic treated unit as a weighted average of untreated donor units. The weights (non-negative, summing to 1) are chosen so the synthetic unit tracks the treated unit's outcome as closely as possible before treatment. After the intervention, the gap between the real treated line and its synthetic twin is the estimated effect. A tight pre-treatment fit is the whole credibility test - if the synthetic cannot match the treated region before anything happened, you have no reason to trust it afterward.

Lumen has 9 US regions in the geo field and 52 weeks of pre-period spend_weekly revenue. We treat a subset with a CTV burst and hold the rest out as the donor pool. First, shape the data into a region-by-week revenue panel.

import pandas as pd, numpy as np

sw = pd.read_parquet("lumen_spend_weekly.parquet")   # week, channel, geo, revenue, ...
panel = (sw.groupby(["week", "geo"], as_index=False)["revenue"].sum()
           .pivot(index="week", columns="geo", values="revenue")
           .sort_index())

INTERVENTION = pd.Timestamp("2026-04-06")   # CTV burst goes live
pre  = panel.loc[panel.index <  INTERVENTION]   # 52 pre-period weeks
post = panel.loc[panel.index >= INTERVENTION]
print(panel.shape, "regions:", list(panel.columns))

treated = "US-CA"                                  # got the CTV burst
adjacent = ["US-NV"]                               # shares a media market -> contaminated
donors = [g for g in panel.columns
          if g not in [treated] + adjacent]         # clean donor pool
print("treated:", treated, "| donors:", donors)

# sanity: donors must be UNAFFECTED by the burst (no spillover) - SUTVA at the geo level

Two credible ways to build the counterfactual. Fit CausalImpact (a Bayesian structural time-series forecast from the donors) and a classic synthetic control with pysyncon, then compare their estimated lift and interval.

from causalimpact import CausalImpact   # tfcausalimpact / pycausalimpact

# response = treated region first, then donor covariates
data = panel[[treated] + donors].copy()
pre_period  = [str(panel.index.min().date()), "2026-03-30"]
post_period = ["2026-04-06", str(panel.index.max().date())]

ci = CausalImpact(data, pre_period, post_period)
print(ci.summary())            # abs + relative effect, 95% credible interval
# ci.plot()  -> observed vs counterfactual, pointwise gap, cumulative lift

from pysyncon import Dataprep, Synth

long = panel.reset_index().melt(id_vars="week", var_name="geo", value_name="revenue")
prep = Dataprep(
    foo=long, predictors=["revenue"], predictors_op="mean",
    dependent="revenue", unit_variable="geo", time_variable="week",
    treatment_identifier=treated, controls_identifier=donors,
    time_predictors_prior=list(pre.index), time_optimize_ssr=list(pre.index),
)
synth = Synth(); synth.fit(dataprep=prep)
print(synth.weights())                     # donor weights, sum to 1
att = synth.att(time_period=list(post.index))
print("estimated ATT (weekly lift):", att)  # gap between real and synthetic

An SCM estimate is only as good as its checks. Confirm the pre-period fit is tight, then run the placebo test: treat each donor as if it were the treated unit and see whether the real effect is extreme against that distribution.

def rmspe(actual, synthetic):
    return np.sqrt(np.mean((actual - synthetic) ** 2))

pre_fit = rmspe(pre[treated].values, synth_pre_values)   # from the fitted model
print(f"pre-period RMSPE = {pre_fit:.1f}")
# rule of thumb: pre-RMSPE should be small vs the revenue scale.
# if it is large, the synthetic never matched -> do NOT trust the post gap

placebo_gaps = {}
for g in donors:
    fake_donors = [d for d in donors if d != g]
    gap = fit_scm_gap(panel, treated=g, donors=fake_donors,   # your demo-2 fit, reused
                      pre=pre.index, post=post.index)
    placebo_gaps[g] = gap                                      # post-period effect if "treated"

real_gap = fit_scm_gap(panel, treated=treated, donors=donors,
                       pre=pre.index, post=post.index)

# ratio of post- to pre-period RMSPE puts every unit on one scale
ranks = sorted([real_gap] + list(placebo_gaps.values()), reverse=True)
p_perm = (ranks.index(real_gap) + 1) / (len(ranks))
print(f"permutation p ≈ {p_perm:.3f}  (real effect should sit in the tail)")

• Swap the donor pool. Drop your two highest-weight donors and refit. Watch the estimate and the pre-period fit move - this is how sensitive SCM is to the pool, and why donor choice is a decision you must defend.
• Placebo on an untreated region. Pick a donor that got no burst, treat it as the treated unit, and run the full pipeline. The effect should be near zero - if it is not, your model is over-fitting.
• Compare the two engines. Put the CausalImpact cumulative lift and the pysyncon ATT side by side. Where they disagree, explain why (BSTS priors vs convex donor weights).
• Optional: re-run with a deliberately contaminated donor (add back the adjacent region) and show how spillover shrinks the measured effect - the SUTVA violation, quantified.

Geo + synthetic control is a genuine gap on every platform we surveyed - we teach it here from the primary sources. The deep econometric proofs stay in the papers and the Udemy econometrics course.