Why a "significant" result can still be garbage
In b1 randomization rescued a clean answer; in b2 you sized the test. But between the design and the dashboard sits a pile of plumbing that quietly decides whether your randomization actually held. Did you split by user or by session? Did one arm accidentally get 52% of the traffic? Does the pipeline even produce honest p-values when there is truly nothing to find? Session 3 makes each of these a runnable check on Lumen's assignment log, so that when a test does come back significant, you have earned the right to believe it.
scipy.stats.chisquare that halts analysis when allocation drifts from intended. Run these before you read a metric, every single time.
Randomize the unit you analyze 6 min live
The single most common validity bug is randomizing by the wrong unit. If you analyze results per user but assign per session, the same person can land in both arms across visits - contaminating the comparison and breaking the independence your test assumes. The rule is blunt: randomize by the unit you analyze by. For a checkout-conversion test where a shopper visits several times, that unit is the user, not the session.
LiveSUTVA - the assumption that quietly breaks3 min▶
SUTVA (stable unit treatment value assumption, Rubin 1980) has two halves, and both matter for online tests:
- No interference - one unit's treatment does not affect another unit's outcome. Referral features, shared inventory, and social feeds break this: a treated user can change an untreated user's behavior.
- No hidden versions - "the treatment" is one well-defined thing. If the new layout renders differently on old browsers, you are secretly running several treatments and averaging them.
Lumen tests a "refer a friend" banner. Treated users invite control users, whose conversion then rises - so the control arm is no longer a clean counterfactual, and the measured effect is diluted. When interference is structural, you switch to cluster randomization (assign whole regions or social clusters) rather than individual users. We build the geo version of this in b8.
Self-studyInternal vs external validity - two different jobs2 min read▶
Keep two ideas separate. Randomization (how you assign the units you have) buys internal validity - the effect you measure is caused by the treatment, not by confounding. Random sampling (how you drew the units from the population) buys external validity - the effect generalizes beyond your sample. An A/B test on logged-in Lumen shoppers can be internally airtight and still not generalize to first-time anonymous visitors. Both matter; they are not the same guarantee.
Stratify, prove, and monitor 4 min live
Plain randomization balances arms in expectation, but any single draw can still be lopsided on an important covariate. Three practices tighten the guarantee: stratified (block) randomization balances within each stratum so arms match on device from the start; the A/A test proves your whole pipeline is honest before you trust an A/B; and the SRM check watches the allocation itself for drift while the test runs.
LiveWhy SRM alarms at p < 0.001, not 0.053 min▶
A sample ratio mismatch is when the observed split deviates from what you intended (a 50/50 design that lands 52/48 on large N). It is a symptom, not a metric - a signal that assignment, logging, or filtering is broken, so the arms are no longer comparable and no downstream result can be trusted.
- Test it with a chi-square goodness-of-fit against the intended ratio.
- The alarm threshold is ~p < 0.001, deliberately far stricter than the 0.05 you would use for a metric - SRM is a rare-but-catastrophic bug, and you do not want a false alarm every twentieth healthy test.
- When SRM fires: stop, do not read the results, and debug the pipeline. Common causes are a redirect that drops one arm, a bot filter applied unevenly, or a broken hash bucket.
Self-studyWhat a clean A/A test looks like2 min read▶
An A/A test runs two identical arms - no real difference - through the exact pipeline you will use for the A/B. If the plumbing is honest, the p-values across many repeats are uniformly distributed on 0 to 1, and about 5% fall below 0.05 purely by chance. That 5% is not a bug - it is alpha doing exactly its job. If instead 15% come back "significant", or the p-values pile up near zero, your pipeline has a leak (often a randomization or logging fault) and any A/B built on it is suspect.
Assign 20,000 users, then stratify by device ★ 8 min · everyone
Start with plain random assignment of 20,000 Lumen users in numpy, then upgrade to stratified randomization so the arms balance on device (mobile / desktop / tablet). Verify the balance both ways so you can feel what stratification buys.
Build the user frame with a device mix, then do a plain coin-flip assignment:
import numpy as np, pandas as pd
rng = np.random.default_rng(7)
N = 20_000
device = rng.choice(["mobile", "desktop", "tablet"],
size=N, p=[0.62, 0.30, 0.08])
users = pd.DataFrame({"user_id": np.arange(N), "device": device})
# plain randomization: independent coin flip per user
users["arm_plain"] = np.where(rng.random(N) < 0.5, "variant", "control")
Now stratify: split within each device group so the arms match on device by construction:
def stratified_assign(frame, strat_col, p=0.5, seed=7):
rng = np.random.default_rng(seed)
arm = np.empty(len(frame), dtype=object)
for _, idx in frame.groupby(strat_col).groups.items():
idx = np.array(idx)
rng.shuffle(idx) # shuffle within stratum
cut = int(round(len(idx) * p))
arm[idx[:cut]] = "variant"
arm[idx[cut:]] = "control"
return arm
users["arm_strat"] = stratified_assign(users, "device")
Verify balance. Compare the device mix across arms under each scheme - stratification pins it near-perfectly:
for col in ["arm_plain", "arm_strat"]:
tab = (users.groupby(col)["device"]
.value_counts(normalize=True)
.unstack().round(3))
print(col, "\n", tab, "\n")
# arm_plain: device shares wobble a little between arms
# arm_strat: device shares are essentially identical across arms
Run an A/A test and read the p-value histogram ★ 8 min · everyone
Prove the pipeline is honest. Simulate two arms with the same true rate (3.2%), run the two-proportion test many times, and confirm the p-values are ~uniform and ~5% land below 0.05. That 5% is alpha, working exactly as designed.
Write one A/A trial: draw both arms from the same rate, run a two-proportion z-test, return the p-value:
import numpy as np
from statsmodels.stats.proportion import proportions_ztest
rng = np.random.default_rng(0)
BASE = 0.032 # identical rate in BOTH arms - no real effect
N_ARM = 20_000
def aa_trial():
a = rng.random(N_ARM) < BASE # arm A conversions
b = rng.random(N_ARM) < BASE # arm B conversions, same rate
count = np.array([a.sum(), b.sum()])
nobs = np.array([N_ARM, N_ARM])
_, p = proportions_ztest(count, nobs)
return p
Repeat it thousands of times and measure the false-positive rate - it should sit right around alpha:
pvals = np.array([aa_trial() for _ in range(5_000)])
false_pos = (pvals < 0.05).mean()
print(f"share of p < 0.05 = {false_pos:.3f}") # ~0.05, as designed
Plot the histogram - a flat bar chart is the signature of a healthy pipeline:
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(pvals, bins=20, range=(0, 1), color="#4F46E5", edgecolor="white")
ax.axhline(len(pvals) / 20, color="#84CC16", linestyle="--",
label="uniform expectation")
ax.set_xlabel("p-value under A/A (no true effect)")
ax.set_ylabel("count")
ax.set_title("A/A p-values are ~uniform - the pipeline is honest")
ax.legend(); fig.tight_layout()
# a flat histogram = trustworthy plumbing; a spike near 0 = a leak
Teams run a scheduled A/A on live traffic every quarter. If it ever comes back with 12% of metrics "significant", someone changed the assignment service, a caching layer, or a logging join - and every A/B since is suspect. The A/A is the smoke detector you hope never goes off.
Catch a sample ratio mismatch with chi-square ★ 6 min · everyone
The last guard. Feed the canonical Lumen SRM example - 20,000 assignments landing 10,420 / 9,580 - to a chi-square goodness-of-fit test, watch the p-value collapse toward zero, and compare it to a clean 50/50. Then write the guard that halts analysis when SRM fires.
Run the chi-square test against the intended 50/50 split for both a broken and a clean allocation:
import numpy as np
from scipy.stats import chisquare
def srm_pvalue(observed, expected_ratio=(0.5, 0.5)):
observed = np.array(observed, dtype=float)
total = observed.sum()
expected = total * np.array(expected_ratio)
# goodness-of-fit: observed counts vs intended split
_, p = chisquare(f_obs=observed, f_exp=expected)
return p
srm = srm_pvalue([10_420, 9_580]) # the canon broken split
clean = srm_pvalue([10_005, 9_995]) # a healthy 50/50
print(f"SRM example p = {srm:.2e}") # ~1e-8, catastrophic
print(f"clean split p = {clean:.3f}") # large, no problem
Wrap it in a guard that stops the analysis before anyone reads a metric - alarm at the strict p < 0.001:
SRM_ALARM = 1e-3 # strict, NOT 0.05 - SRM is rare but catastrophic
def guard_or_stop(observed):
p = srm_pvalue(observed)
if p < SRM_ALARM:
raise RuntimeError(
f"SRM detected (p={p:.2e}). Split is {observed}, "
"expected ~50/50. STOP - debug the pipeline, "
"do not read the results.")
print(f"allocation OK (p={p:.3f}) - safe to analyze")
guard_or_stop([10_005, 9_995]) # passes
guard_or_stop([10_420, 9_580]) # raises -> analysis halts
Wire guard_or_stop as the first line of your analysis script. A test with SRM is not a weaker result - it is a non-result. Reading its conversion lift is like trusting a scale you know is miscalibrated.
This week ◐ 40 min total
- Add a novelty-effect check. Split the A/A (or a real A/B) outcome by day-since-start and plot the arm difference over time. A gap that shrinks as the days pass is a novelty / primacy signal, not a durable effect - decide how long to run before you trust the number.
- Stratify by a second covariate. Extend
stratified_assignto block on device and new-vs-returning at once (cross the strata), and confirm both stay balanced across arms. - Break your own pipeline. Introduce a bug - drop 4% of one arm's rows before the test - then confirm your A/A false-positive rate climbs and your SRM guard fires. Feel both alarms trip.
- Optional: extend the SRM guard to a 3-arm test (e.g. 34/33/33 intended) and check the chi-square degrees of freedom update correctly.
Three questions before you go 🎯 ◐ 90 seconds
1 · You analyze conversion per user but assign treatment per session. The core problem is...
Randomize by the unit you analyze by. Session-level assignment lets one user land in both arms across visits, breaking independence and mixing the arms. For a user-level metric, assign per user with a sticky hash.
2 · A clean A/A test on your pipeline should produce...
With no true effect, an honest pipeline emits uniform p-values, so ~5% fall under 0.05 - that is alpha working as designed. A spike near zero or a much higher rate means the plumbing has a leak.
3 · Your 50/50 test logged 10,420 / 9,580 and the chi-square p is ~1e-8. You should...
That is a sample ratio mismatch. A p below the ~0.001 alarm threshold means allocation deviated from intended, so the arms are not comparable and no downstream metric can be trusted. Stop and fix the assignment or logging.
What this session covers
Validity threats and trust checks are scattered across the A/B and design courses; we build the working guards here and point to the sources for the fuller taxonomy.