Free power hiding in your history table
Every Lumen shopper who lands in your test already has a past: how much they spent in the prior 30 days, how often they visited. That history is a strong predictor of what they will do next - and, crucially, it was fixed before you flipped anyone into treatment. CUPED (Controlled-experiment Using Pre-Existing Data) uses that predictable-in-advance part to de-noise the outcome. You are not changing the effect you estimate; you are removing variation that has nothing to do with the treatment. Less noise means tighter confidence intervals, which means you need fewer users to reach the same power. For Lumen's canonical test, a pre-period correlation of ρ≈0.6 turns ~32,000/arm into ~20,500/arm.
Subtract the predictable part of the outcome 6 min live
Take the outcome Y (say, spend during the test) and a pre-period covariate X (prior-30-day spend). Define the CUPED-adjusted outcome Y_cuped = Y - θ(X - E[X]), with θ = Cov(Y,X)/Var(X). Because we subtract only the mean-centred covariate, the expected value of Y_cuped is unchanged - so the estimated treatment effect is identical in expectation. But its variance is multiplied by (1 - ρ²), where ρ = corr(Y,X). The variance reduction is exactly ρ². A pre-period version of the same metric commonly carries ρ≈0.5-0.7, so 25-50% of the noise simply vanishes.
LiveWhy it is unbiased - the pre-treatment guarantee3 min▶
The whole thing rests on one fact: the covariate is measured before treatment, so it cannot be affected by treatment. That makes X independent of assignment, so subtracting θ(X - E[X]) is subtracting the same expected quantity from both arms. The difference in means - the treatment effect - is untouched in expectation. You are only cancelling noise that both arms share.
- θ = Cov(Y,X)/Var(X) - the OLS slope of Y on X; the best linear amount of X to subtract.
- Variance reduction = ρ² where ρ = corr(Y,X). Same-metric pre-period: ρ≈0.5-0.7 → 25-50% cut.
- Fewer users for the same power - required N scales with variance, so a 36% variance cut is a ~36% smaller sample: ~32,000/arm → ~20,500/arm on Lumen.
Microsoft/Bing introduced CUPED (Deng, Xu, Kohavi & Walker, WSDM 2013) and reported it routinely halving the data needed for the same sensitivity. It is now standard at essentially every large experimentation platform - and almost never taught in a course, which is exactly why it is here.
Self-studyCUPED is regression adjustment (Lin 2013)2 min read▶
CUPED with one pre-period covariate is algebraically the same as fitting Y ~ treatment + X by OLS and reading the treatment coefficient. That reframes it as plain regression adjustment. Lin (2013) gives the design-robust version - include the treatment×covariate interaction (equivalently, center X and interact) so the adjustment never hurts precision even under model misspecification, and the usual robust standard errors stay valid. In practice: use OLS with the pre-period covariate, robust SEs, and you have CUPED with a safety net.
The covariate must be strictly pre-treatment 4 min live
There is exactly one fatal mistake with CUPED, and it is tempting because during-experiment covariates are often even more correlated with the outcome. If X is measured during or after treatment, it can itself be affected by the treatment - it becomes a post-treatment variable (a mediator or collider). Adjusting for it reintroduces bias, and now your "de-noised" effect is simply wrong. The scatter below shows the mechanism: a clean pre-period covariate scatters symmetrically around the regression line in both arms; a contaminated one shifts, dragging the adjusted estimate off the true effect.
Self-studyPre vs post - a one-line litmus test2 min read▶
Ask: could this variable have been written down before the user entered the experiment? If yes (prior-30-day spend, tenure, historical visit count) it is safe. If it is anything the user did after assignment (sessions during the test, add-to-cart in the test window) it is post-treatment and off-limits for CUPED, no matter how predictive it looks. The seductiveness of a high correlation is exactly the trap.
Compute θ, form Y_cuped, measure the variance cut ★ 8 min · everyone
Build a Lumen population where each user has a pre-period covariate - prior-30-day spend - correlated with their in-test outcome at ρ≈0.6. Compute θ from Cov/Var, form the adjusted outcome, and confirm the variance drops ~36%, which turns the ~32,000/arm test into ~20,500/arm.
Simulate the pre-period covariate and a correlated outcome (ρ≈0.6 by canon), split 50/50 with a real +0.4pp-scale lift baked in:
import numpy as np, pandas as pd
rng = np.random.default_rng(42)
N = 40_000
RHO = 0.6 # canon: pre-period spend corr ~0.6
X = rng.normal(60, 20, N) # prior-30-day spend (PRE-treatment)
noise = rng.normal(0, 20, N)
# outcome Y correlated with X at ~RHO; treatment adds a small lift
T = (rng.random(N) < 0.5).astype(int)
Y = 50 + RHO * (X - 60) + np.sqrt(1 - RHO**2) * noise + 2.0 * T
df = pd.DataFrame({"X": X, "Y": Y, "T": T})
θ is the covariance of Y and X over the variance of X (pooled across arms), and Y_cuped subtracts the mean-centred covariate:
theta = np.cov(df.Y, df.X, ddof=1)[0, 1] / np.var(df.X, ddof=1)
df["Y_cuped"] = df.Y - theta * (df.X - df.X.mean())
var_raw = df.Y.var(ddof=1)
var_cuped = df.Y_cuped.var(ddof=1)
print(f"theta = {theta:.3f}")
print(f"variance reduction = {1 - var_cuped/var_raw:.1%}") # ~36% (= rho^2)
Turn the variance cut into a sample-size cut - required N scales with variance, so a 36% variance reduction is a 36% smaller test:
reduction = 1 - var_cuped / var_raw # ~0.36
n_fixed = 32_000 # per arm, from b2
n_cuped = round(n_fixed * (1 - reduction), -2)
print(f"~{n_fixed:,}/arm -> ~{int(n_cuped):,}/arm") # ~32,000 -> ~20,500
The mean of Y_cuped matches the mean of Y - the effect is preserved - but its spread is a third smaller. Same power, ~35% fewer users, no new assumptions. That is the whole win.
Prove CUPED equals regression adjustment ★ 7 min · everyone
CUPED is not a separate trick - it is OLS of the outcome on treatment plus the pre-period covariate. Estimate the effect three ways (raw diff, CUPED-adjusted diff, OLS with X) and watch the two adjusted numbers coincide while the standard error shrinks.
Raw difference vs CUPED-adjusted difference in means:
raw = df.loc[df.T==1, "Y"].mean() - df.loc[df.T==0, "Y"].mean()
cup = df.loc[df.T==1, "Y_cuped"].mean() - df.loc[df.T==0, "Y_cuped"].mean()
print(f"raw diff = {raw:.3f}")
print(f"CUPED diff = {cup:.3f}") # ~same point estimate as raw
Now the OLS of Y on treatment and the pre-period covariate - the treatment coefficient is the CUPED-adjusted effect:
import statsmodels.formula.api as smf
m = smf.ols("Y ~ T + X", data=df).fit()
print(m.params["T"], m.bse["T"]) # coef == CUPED effect; SE is shrunk
m_raw = smf.ols("Y ~ T", data=df).fit()
print(m_raw.params["T"], m_raw.bse["T"]) # bigger SE - no covariate
The T coefficient from Y ~ T + X matches the CUPED difference, and its standard error is markedly smaller than the raw model's. For the design-robust Lin (2013) version, fit Y ~ T * center(X) with robust SEs so the adjustment can never hurt:
df["Xc"] = df.X - df.X.mean()
m_lin = smf.ols("Y ~ T * Xc", data=df).fit(cov_type="HC1")
print(m_lin.params["T"], m_lin.bse["T"]) # design-robust CUPED
The pitfall demo - watch a post-treatment covariate lie ★ 7 min · everyone
Now break it on purpose. Replace the clean pre-period covariate with one measured during the test - one that responds to treatment - and watch the CUPED-adjusted effect drift away from the truth. This is the single mistake that ruins CUPED, and seeing it once inoculates you for good.
Manufacture a contaminated covariate: an in-test signal that treatment itself moves (so it carries treatment information):
TRUE_EFFECT = 2.0 # what we baked into Y via +2.0*T # BAD covariate: measured DURING the test, pushed up by treatment X_post = 0.5 * df.X + 8.0 * df.T + rng.normal(0, 10, N) theta_bad = np.cov(df.Y, X_post, ddof=1)[0,1] / np.var(X_post, ddof=1) df["Y_bad"] = df.Y - theta_bad * (X_post - X_post.mean())
Estimate the effect with the contaminated adjustment and compare to the truth and to the clean CUPED number:
bad = df.loc[df.T==1, "Y_bad"].mean() - df.loc[df.T==0, "Y_bad"].mean()
print(f"true effect = {TRUE_EFFECT:.2f}")
print(f"clean CUPED effect = {cup:.2f}") # ~2.0, unbiased
print(f"post-covariate est = {bad:.2f}") # BIASED - pulled off 2.0
# adjusting for something treatment moved subtracts real effect => bias
The clean CUPED estimate sits on the true 2.0; the post-treatment one is visibly off. By adjusting for a variable that treatment itself changed, you subtracted part of the real effect. This is why the pre-treatment rule is non-negotiable - no correlation is worth a biased answer.
This week ◐ 40 min total
- Sweep ρ, plot variance-reduction = ρ². Regenerate the population for ρ from 0.1 to 0.9, compute the realized variance reduction each time, and plot it against ρ². Confirm the points land on the ρ² curve - the theory is exact.
- Try a visit-count covariate. Swap prior-30-day spend for prior-30-day visit count (a weaker predictor, ρ≈0.3-0.4). Recompute θ and the sample-size saving, and see how the win shrinks with the correlation.
- Confirm the effect is preserved. Across many seeds, plot the CUPED effect estimate vs the raw estimate. Both center on the true effect; only the CUPED one has a tighter spread.
- Optional: combine two pre-period covariates (spend + visits) via the multivariate regression-adjustment form and see whether the extra covariate buys meaningful additional variance reduction.
Three questions before you go 🎯 ◐ 90 seconds
1 · Why does CUPED reduce variance without introducing bias?
A pre-treatment covariate cannot be affected by treatment, so mean-centred subtraction removes shared noise from both arms equally. The expected difference in means - the effect - is unchanged; only the variance drops, by ρ².
2 · With θ = Cov(Y,X)/Var(X) and ρ = corr(Y,X), the variance reduction from CUPED is...
Var(Y_cuped) = Var(Y)(1 - ρ²), so the reduction is exactly ρ². Required N scales with variance, so a 36% variance cut is a ~36% smaller sample.
3 · What is the one fatal mistake with CUPED?
The covariate must be strictly pre-treatment. A during/after covariate is post-treatment (a mediator/collider); adjusting for it subtracts part of the real effect and biases the estimate - no matter how predictive it looks.
What this session covers
CUPED is standard practice at every large experimentation platform and taught by essentially none of them - a genuine gap this course owns. We teach it from the primary source: Deng, Xu, Kohavi & Walker, "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data," WSDM 2013 (Microsoft/Bing), with the regression-adjustment view from Lin (2013).