Sequential testing, and why peeking quietly breaks your A/B test

In b2 you computed that Lumen's flagship test - control 3.2% vs variant 3.6% - needs ~32,000 users per arm to detect that 0.4pp lift at alpha 0.05, power 0.80. That number is a promise: the p-value you read is only a valid 5% test at that N. But real dashboards update live, and the temptation is unbearable - you look every morning, and the day it crosses 0.05 you call it. That habit, "peeking," silently converts your 5% false-positive rate into 20-30%+. Today you see it happen on a simulator, understand precisely why, and build the three families of fix that let you monitor continuously and still be honest.

This simulator runs many A/A tests - two arms with no real difference at all, seeded from Lumen's 3.2% baseline and its ~32,000/arm horizon. The honest false-positive rate is 5% (the dashed line): that is what you get if you look exactly once, at the pre-committed N. The other number is what happens when you peek along the way and stop the first time p dips under 0.05. There is no real effect anywhere in this data - every "significant" result the peeking column reports is manufactured by looking.

A classical p-value is calibrated for a single decision at a pre-committed sample size. Under the null, the running test statistic wanders like a random walk. Give it many chances to cross the 0.05 threshold - one per peek - and the probability that it crosses at least once grows far past 5%. This is a multiple-comparisons problem in disguise, except the comparisons are strung out over time instead of across metrics. You are not testing once; you are testing every day and reporting the best day.

There are three respectable answers. The first two keep clean inference (a valid p-value or interval at any stopping time); the third abandons fixed inference on purpose to chase a different goal - minimizing lost conversions. Know which problem you are solving before you pick.

Rebuild the widget's result yourself. Simulate many A/A tests on Lumen's 3.2% baseline - no real difference - accumulate users day by day, and count how often a daily peek finds "significance" versus a single look at the final N. You should land ~5% for the single look and ~20-30% for peeking.

import numpy as np
from scipy import stats
rng = np.random.default_rng(42)

P0 = 0.032           # Lumen baseline; SAME in both arms (a true A/A)
PER_DAY = 2_000      # users per arm per day (50/50 of ~4,000/day)
DAYS = 16            # ~32,000/arm final horizon
RUNS = 4_000

def one_trial():
    a_succ = a_n = b_succ = b_n = 0
    hit_peek = False
    for _ in range(DAYS):
        a_succ += rng.binomial(PER_DAY, P0); a_n += PER_DAY
        b_succ += rng.binomial(PER_DAY, P0); b_n += PER_DAY
        if two_prop_p(a_succ, a_n, b_succ, b_n) < 0.05:
            hit_peek = True                    # stop-early rule fired at least once
    p_final = two_prop_p(a_succ, a_n, b_succ, b_n)   # the ONE valid look
    return hit_peek, (p_final < 0.05)

def two_prop_p(x1, n1, x2, n2):
    p1, p2 = x1 / n1, x2 / n2
    p = (x1 + x2) / (n1 + n2)                  # pooled rate under H0
    se = np.sqrt(p * (1 - p) * (1 / n1 + 1 / n2))
    if se == 0: return 1.0
    z = (p1 - p2) / se
    return 2 * (1 - stats.norm.cdf(abs(z)))    # two-sided

res = np.array([one_trial() for _ in range(RUNS)])
peek_fpr  = res[:, 0].mean()      # "significant on ANY day"
final_fpr = res[:, 1].mean()      # significant at the pre-committed N only
print(f"peeking daily  -> false positives {peek_fpr:.1%}")   # ~20-30%
print(f"single final look -> false positives {final_fpr:.1%}") # ~5%

Now implement a simple mSPRT-style always-valid statistic and rerun the exact same daily-peeking loop. This time the false-positive rate stays near 5% even though you look every single day - the boundary was built to be valid at all times.

def msprt_pvalue(diff, n_eff, tau2, sigma2):
    # diff = observed rate difference; n_eff = harmonic effective N;
    # tau2 = mixing-prior variance on the true effect; sigma2 = per-obs variance
    v = sigma2 / n_eff                         # variance of the estimate
    # mixture likelihood ratio of H1 (effect ~ N(0, tau2)) vs H0 (effect = 0)
    lr = np.sqrt(v / (v + tau2)) * np.exp(
        (diff ** 2) / 2 * (tau2 / (v * (v + tau2))))
    return min(1.0, 1.0 / lr)                  # always-valid p-value

def one_av_trial(tau2=0.001):
    a_succ = a_n = b_succ = b_n = 0
    fired = False
    for _ in range(DAYS):
        a_succ += rng.binomial(PER_DAY, P0); a_n += PER_DAY
        b_succ += rng.binomial(PER_DAY, P0); b_n += PER_DAY
        diff = a_succ / a_n - b_succ / b_n
        n_eff = 1 / (1 / a_n + 1 / b_n)
        sigma2 = P0 * (1 - P0)
        if msprt_pvalue(diff, n_eff, tau2, sigma2) < 0.05:
            fired = True
    return fired

av_fpr = np.mean([one_av_trial() for _ in range(RUNS)])
print(f"always-valid, peeked daily -> false positives {av_fpr:.1%}")  # ~5%

Code a 2-arm Thompson-sampling bandit on a Lumen test where the variant genuinely converts better (3.2% vs 3.6%). Watch it steer traffic toward the winner, minimizing regret - then see why the same behaviour makes a clean fixed p-value impossible.

TRUE = {"control": 0.032, "variant": 0.036}    # variant really is better
N = 32_000
alpha = {"control": 1.0, "variant": 1.0}       # Beta priors
beta  = {"control": 1.0, "variant": 1.0}
pulls = {"control": 0, "variant": 0}

for _ in range(N):
    draw = {a: rng.beta(alpha[a], beta[a]) for a in TRUE}
    arm = max(draw, key=draw.get)              # serve the best sample
    reward = rng.random() < TRUE[arm]          # did they convert?
    pulls[arm] += 1
    alpha[arm] += reward
    beta[arm]  += 1 - reward

best = max(TRUE.values())
got  = sum(TRUE[a] * pulls[a] for a in TRUE)   # expected conversions earned
regret = best * N - got
print(pulls)                                   # most traffic went to 'variant'
print(f"expected regret over {N} users: {regret:.1f} conversions")
# compare: a 50/50 fixed A/B "wastes" ~half the traffic on the weaker arm

• Plot false-positive rate vs number of peeks. Take demo 1 and vary the peeking cadence - look once, twice, weekly, daily, hourly. Plot the false-positive rate against the number of looks and watch it climb from 5% toward 30%+.
• Try a group-sequential design. Pick 5 pre-planned looks and apply O'Brien-Fleming-style boundaries (spend little alpha early, most at the end). Confirm the family-wise error stays at 5% while still allowing an early stop on a strong effect.
• Measure the always-valid tax. Rerun demo 2 with a genuine 3.2% vs 3.6% effect and record how many users the always-valid test needs to detect it, versus the ~32,000/arm fixed design. Quantify the conservatism.
• Optional: add a third arm to the Thompson bandit and plot cumulative regret over time for bandit vs 50/50 vs a group-sequential test - three curves, one picture of the tradeoff.

Sequential testing and the peeking problem are a genuine gap on every major platform (DeepLearning.AI, Coursera, 365, LinkedIn, Udemy). This session owns it, taught from the primary source - Johari, Pekelis, Walsh & Koomen, "Peeking at A/B Tests," KDD 2017. The bandit depth lives with Lazy Programmer's Bayesian A/B Udemy course.