Potential outcomes, and the estimand you can never fully see

Lumen's CMO wants to know one thing: "If we ship the new product-page layout, will more people check out?" That sounds like a reporting question, but it is a causal one. It compares the world where we ship the change to the world where we do not - for the same shoppers, at the same time. Only one of those worlds is ever real. Session 1 gives you the framework that makes this precise (potential outcomes), names the thing you are actually estimating (the estimand), and shows in code why a plain before/after comparison quietly lies while a randomized one does not.

The Neyman-Rubin framework says: for each unit i there are two potential outcomes - Y_i(1), what happens if it gets the treatment, and Y_i(0), what happens if it does not. The causal effect for that unit is Y_i(1) - Y_i(0). The catch is brutal and permanent: you can only ever observe one of the two, because a shopper either sees the new layout or the old one, never both. The other is the missing counterfactual.

If you let shoppers self-select into the new layout (say, only logged-in loyalty members see it), then treatment and outcome share a common cause - engagement. Engaged people convert more and are likelier to be treated. That common cause is a confounder, and it opens a "backdoor path" that makes the naive difference a blend of real effect plus selection. Randomization severs that path by making assignment independent of everything - observed and unobserved.

Every experiment in this course starts by deciding how many users it needs. Here is the finished planner - drag the dials and feel how baseline rate, the lift you want to detect, and your error tolerances trade off against sample size and runtime. In Builder Session 2 you rebuild this math from scratch in statsmodels.

In real life the counterfactual is missing. In a simulation, we play god: we generate Y(1) and Y(0) for every synthetic Lumen shopper, so we know the true effect and can check whether our estimators recover it.

import numpy as np, pandas as pd
rng = np.random.default_rng(42)

N = 20_000
engagement = rng.beta(2, 5, N)            # skewed 0..1, most shoppers low

# TRUE potential outcomes (we are omniscient here):
# baseline buy-prob rises with engagement; the new layout adds a flat +2pp lift
base = 0.02 + 0.10 * engagement           # Y(0) probability
p0   = base
p1   = np.clip(base + 0.02, 0, 1)          # Y(1) probability = +2pp TRUE effect

Y0 = (rng.random(N) < p0).astype(int)     # outcome if OLD layout
Y1 = (rng.random(N) < p1).astype(int)     # outcome if NEW layout
df = pd.DataFrame({"engagement": engagement, "Y0": Y0, "Y1": Y1})

true_ate = (df.Y1 - df.Y0).mean()
print(f"True ATE = {true_ate:.4f}")   # ~0.020 by construction (the +2pp)

Now assign treatment the way it usually happens in the wild: the most engaged shoppers opt into the new experience. Reveal only the observed outcome for each, and compare group means. The naive number will miss the true ATE - on purpose.

# self-selection: engaged shoppers are far likelier to get the new layout
p_treat = 0.15 + 0.70 * df.engagement
T = (rng.random(N) < p_treat).astype(int)

# reveal only the OBSERVED outcome (the fundamental problem in code)
Y_obs = np.where(T == 1, df.Y1, df.Y0)
df["T"], df["Y"] = T, Y_obs

naive = df.loc[df.T==1, "Y"].mean() - df.loc[df.T==0, "Y"].mean()
print(f"Naive diff = {naive:.4f}  vs true ATE {true_ate:.4f}")
# naive is INFLATED - treated group was more engaged to begin with

Change one line - assign by coin flip instead of by engagement - and the same naive difference now lands on the true ATE. Nothing else changed. That is the entire argument for controlled experiments in three lines of Python.

T_rand = (rng.random(N) < 0.5).astype(int)          # 🎲 independent of everything
Y_rand = np.where(T_rand == 1, df.Y1, df.Y0)

est = Y_rand[T_rand==1].mean() - Y_rand[T_rand==0].mean()
print(f"Randomized estimate = {est:.4f}  vs true ATE {true_ate:.4f}")
# now they match within sampling noise - bias is gone

• Run all three build-alongs and keep the notebook - b2 through b10 reuse this simulate-and-check pattern to sanity-test every estimator against a known truth.
• Sweep the confounding strength. Change the 0.70 * engagement coefficient in demo 2 to 0.1, 0.4, 0.9 and plot naive-bias vs confounding. See the bias grow smoothly with selection.
• Write the ATT and a CATE. Compute the effect among the treated, and the effect for high-engagement (engagement > 0.5) shoppers. Confirm ATE = ATT under randomization but not under self-selection.
• Optional: add a second, unobserved confounder and show that dropping the observed one from an adjustment still leaves bias - motivating why randomization beats "just control for it".

The potential-outcomes framework is the shared foundation of every causal course. We teach ~80% of the working core here; the deep proofs and the graphical (Pearlian) treatment live in the sources below.