Power and sample size, or how big does this test actually need to be?

Lumen wants to test a new "hero-first" product-page layout, hoping to lift checkout conversion from a baseline of 3.2%. Before the experiment runs, someone will ask: "How long do we leave it on?" The honest answer is a calculation, not a vibe. Too few users and a real win hides inside the noise - you kill a good idea. Too many and you burn traffic and calendar you could have spent on the next test. Session 2 turns that trade into arithmetic: pick your tolerance for two kinds of mistake, pick the smallest effect worth detecting, and the sample size falls out. We derive it, then compute it in code, then convert it into a runtime in days.

Every test can be wrong in exactly two ways. A Type I error (false positive) is calling a win when nothing happened - its rate is alpha, conventionally 0.05. A Type II error (false negative) is missing a real effect - its rate is beta. Power = 1 - beta is the probability you detect an effect that is genuinely there, conventionally set to 0.80. These four cells are the whole design problem: you buy lower error rates with more sample.

The minimum detectable effect is the smallest lift you care about catching. It is an input you decide with the business - "a 0.4pp absolute lift, from 3.2% to 3.6%, is worth shipping" - and it drives everything. Smaller MDE means you are hunting a fainter signal, which costs sample fast: sample size scales with 1 / delta^2, so halving the MDE roughly quadruples the test. Get the product to commit to an MDE before you compute anything.

You met this planner in b1 as a preview. Today it is the whole point. Drag the dials: raise the baseline, shrink the MDE, tighten alpha, push power up - and watch sample size and runtime respond. It is seeded with Lumen's flagship test (3.2% baseline, +12.5% relative MDE, alpha 5%, power 80%, 4,000 users/day). The three build-alongs below rebuild this tool's math by hand so nothing here is a black box.

Rebuild the planner's core number. We use statsmodels.stats.power with NormalIndPower and turn Lumen's two rates into an effect size with proportion_effectsize. The target: ~32,000 per arm for the 3.2% → 3.6% test - and we state every assumption alongside it.

import numpy as np
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

p1, p2 = 0.032, 0.036          # Lumen: control 3.2% vs variant 3.6%
alpha, power = 0.05, 0.80       # conventions, chosen up front
# arcsine-transformed standardized effect for a two-proportion test
effect = proportion_effectsize(p2, p1)

analysis = NormalIndPower()
n_per_arm = analysis.solve_power(
    effect_size=effect,
    alpha=alpha,
    power=power,
    ratio=1.0,                  # 50/50 split, equal arms
    alternative="two-sided",    # we care about up OR down
)
print(f"n per arm  = {np.ceil(n_per_arm):,.0f}")   # ~32,000
print(f"total N    = {np.ceil(n_per_arm)*2:,.0f}") # ~64,000

p, delta = 0.032, 0.004                 # baseline and absolute MDE
approx = 16 * p * (1 - p) / delta**2    # sigma^2 = p(1-p)
print(f"rule-of-16 ≈ {approx:,.0f} per arm")   # ~31,000, agrees

A single number hides the shape of the decision. Sweep the sample size, compute power at each point, and plot it. You will see power climb toward 1.0 with diminishing returns - and read straight off the curve how much traffic buys you how much detection.

import numpy as np
import matplotlib.pyplot as plt
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect = proportion_effectsize(0.036, 0.032)
analysis = NormalIndPower()

ns = np.arange(2_000, 60_001, 2_000)      # sweep sample size per arm
powers = [analysis.solve_power(effect_size=effect, nobs1=n,
                               alpha=0.05, alternative="two-sided")
          for n in ns]

fig, ax = plt.subplots(figsize=(7, 4))
ax.plot(ns, powers, color="#4F46E5", linewidth=2.5)
ax.axhline(0.80, color="#84CC16", linestyle="--", label="power = 0.80")
ax.axvline(32_000, color="#DC2626", linestyle=":", label="~32,000 / arm")
ax.set_xlabel("sample size per arm")
ax.set_ylabel("power (1 - beta)")
ax.set_title("Power vs sample size - Lumen 3.2% to 3.6% test")
ax.legend()
fig.tight_layout()
fig.savefig("power_curve.png", dpi=120)
# the curve rises steeply, then flattens: past ~32k you pay a lot for little

A sample size means nothing to a stakeholder until it becomes a calendar. Wrap the whole calculation in one function that takes the business inputs and returns both the per-arm N and the number of days at Lumen's traffic. At 4,000 eligible users/day split 50/50, the ~64,000-user test runs ~16-17 days.

import numpy as np
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

def plan_test(baseline, mde_rel, alpha=0.05, power=0.80,
              daily_traffic=4000, two_sided=True):
    """Return (n_per_arm, days) for a two-proportion test.
    baseline: control rate, e.g. 0.032
    mde_rel:  relative lift to detect, e.g. 0.125 for +12.5%
    daily_traffic: eligible users/day, split 50/50 across 2 arms
    """
    p1 = baseline
    p2 = baseline * (1 + mde_rel)            # 0.032 -> 0.036
    effect = proportion_effectsize(p2, p1)
    alt = "two-sided" if two_sided else "larger"
    n = NormalIndPower().solve_power(
        effect_size=effect, alpha=alpha, power=power,
        ratio=1.0, alternative=alt)
    n_per_arm = int(np.ceil(n))
    total = n_per_arm * 2
    days = int(np.ceil(total / daily_traffic))   # 50/50 uses all traffic
    return n_per_arm, days

n_arm, days = plan_test(baseline=0.032, mde_rel=0.125,
                        daily_traffic=4000)
print(f"{n_arm:,} per arm  ->  {n_arm*2:,} total  ->  {days} days")
# 32,000 per arm  ->  64,000 total  ->  16 days  (~2.5 weeks)

• Feel the blow-up. Recompute N for a smaller MDE - drop the target lift from +12.5% to +6%. Watch the per-arm sample roughly quadruple. Plot N vs MDE and see the 1/delta^2 curve with your own eyes.
• Power a mean metric too. Lumen also tracks average order value ($92 canonical). Use statsmodels.stats.power.TTestIndPower with a standardized effect (Cohen's d = delta/sigma) to size a t-test, and compare the feel to the proportion case.
• Tighten the rates. Rerun the planner at alpha 0.01 and at power 0.90. Note how much extra sample each buys you, and decide which you would actually pay for on Lumen's flagship test.
• Optional: add an unequal allocation option (ratio = 2.0, more traffic to control) and see how a lopsided split changes the total N for the same power.

Power and sample size are taught across the A/B and statistics courses; we build the working core in statsmodels and leave the deeper theory and the tool UIs to the sources below.