Report the range, not just the point
"Conversion is 3.2%" is a point estimate. "Conversion is 3.2%, 95% CI [2.71%, 3.69%]" is a claim you can defend - and, crucially, a claim that tells you what you cannot yet see. The interval carries the sample size, the noise, and the honesty all at once. Today you compute intervals for means and for proportions, learn when the textbook Wald interval quietly fails, and read the interval correctly - which is harder than it looks.
z vs t, Wald vs Wilson, and what 95% means 7 min live
Which interval you reach for depends on what you know and how much data you have. Four tools, one correct interpretation.
LiveThe four tools, and when each applies3 min▶
- z-interval (sigma known). When you somehow know the population standard deviation, the sampling distribution is normal and you use z critical values (1.96 for 95%). Rare in practice - you almost never know sigma.
- t-interval (sigma unknown, small n). The realistic case: you estimate sigma from the sample, which adds uncertainty, so you use the t distribution. Its fatter tails give a slightly wider interval. As n grows, t converges to z and the difference vanishes.
- Wald vs Wilson (for a proportion). The Wald interval,
p-hat +/- z*sqrt(p-hat(1-p-hat)/n), is the textbook formula - but it behaves badly whenn*pis small (few "successes"), producing intervals that are too narrow or run past 0 or 1. The Wilson interval fixes this and is the better default for rare events. - Bootstrap CI. Resample with replacement, recompute the statistic, and read the 2.5th and 97.5th percentiles. Works for any statistic, no formula needed - the b5 tool, applied.
Build every interval on Lumen's numbers ★ 22 min · everyone builds
A t-interval for mean order value, Wald and Wilson for the 3.2% conversion rate, the canonical [2.71%, 3.69%] result, and a bootstrap CI to tie it all back to b5.
t-interval for the mean. sigma is unknown (we estimate it from the sample), so t is correct.
import numpy as np, pandas as pd
from scipy import stats
df = pd.read_parquet("lumen_orders.parquet")
ov = df["order_value"].to_numpy()
n = ov.size
m = ov.mean() # ~ 74
se = ov.std(ddof=1) / np.sqrt(n) # SE = s / sqrt(n), s ~ 38
lo, hi = stats.t.interval(0.95, df=n - 1, loc=m, scale=se)
print(f"mean order value : {m:.2f}")
print(f"95% t-interval : [{lo:.2f}, {hi:.2f}]") # ~ [73.7, 74.3] at n=50000
# At huge n the interval is razor thin - and t is indistinguishable from z.
Wald vs Wilson for the conversion rate. 3.2% of 5,000 sessions = 160 conversions. Compare the two intervals.
from statsmodels.stats.proportion import proportion_confint
count, nobs = 160, 5000 # 3.2% observed conversion at n = 5000
wald = proportion_confint(count, nobs, alpha=0.05, method='normal')
wilson = proportion_confint(count, nobs, alpha=0.05, method='wilson')
print("Wald :", [round(100*x, 2) for x in wald]) # ~ [2.71, 3.69] %
print("Wilson:", [round(100*x, 2) for x in wilson]) # ~ [2.74, 3.73] %
# At n=5000 with 160 successes they nearly agree. Now shrink the sample...
for nobs_small in (100, 50):
c = round(0.032 * nobs_small)
w = proportion_confint(c, nobs_small, method='normal')
wl = proportion_confint(c, nobs_small, method='wilson')
print(f"n={nobs_small:>3} Wald {[round(100*x,2) for x in w]} "
f"Wilson {[round(100*x,2) for x in wl]}")
# Small n*p: Wald turns too narrow / can dip below 0; Wilson stays sensible.
The canonical result. State the assumptions, then read the punchline out loud.
# Assumptions: n = 5000 sessions, observed conversion 3.2% (160 conversions),
# 95% confidence, Wald (normal) interval.
p, n, z = 0.032, 5000, 1.96
se = (p * (1 - p) / n) ** 0.5 # ~ 0.00249 (0.25pp)
lo, hi = p - z*se, p + z*se
print(f"95% Wald CI: [{100*lo:.2f}%, {100*hi:.2f}%]") # [2.71%, 3.69%]
variant = 0.036
print("variant 3.6% inside the control CI?", lo < variant < hi) # True
# 3.6% sits INSIDE [2.71%, 3.69%] at n=5000 - the data cannot yet tell
# 3.2% and 3.6% apart. That is exactly why experimentation needs ~32,000/arm.
Bootstrap percentile CI. Formula-free, straight from b5 - a sanity check on the t-interval.
rng = np.random.default_rng(42)
B = 2000
boot = np.array([rng.choice(ov, size=ov.size, replace=True).mean()
for _ in range(B)])
lo, hi = np.percentile(boot, [2.5, 97.5])
print("bootstrap 95% CI :", [round(lo, 2), round(hi, 2)]) # ~ [73.7, 74.3]
print("t-interval 95% CI: matches - two roads, one destination")
Here is the bridge to the whole experimentation course. Lumen's control converts at 3.2%; a proposed variant is hoped to hit 3.6%. At n=5,000 the control's 95% CI is [2.71%, 3.69%] - and 3.6% sits comfortably inside it. The interval is telling you, before you spend a cent, that this sample is far too small to separate the two rates. Pushing that overlap apart to a confident "the variant wins" is what forces ~32,000 sessions per arm in the experimentation course. The confidence interval is not a footnote - it is the sample-size calculator in disguise.
This week ◐ 40 min total
- Watch coverage happen. Draw 100 samples of size n=200 from
ov, build a t-interval from each, and count how many contain the true population mean. You should land near 95 - that is the correct interpretation, measured. - Wald breaks, Wilson holds. Simulate n=30 with a true 3.2% rate many times. How often does the Wald interval dip below 0% or miss the truth versus Wilson? Report both coverage rates.
- Shrink the interval. Recompute the conversion Wald CI at n = 5,000, 10,000, and 32,000. At which n does 3.6% finally fall outside the control CI? Connect it to the experimentation handoff.
- Optional: compute the t-interval by hand (
m +/- stats.t.ppf(0.975, n-1) * se) and confirm it matchesstats.t.interval.
Three questions before you go 🎯 ◐ 90 seconds
1 · What does a 95% confidence interval actually mean?
The true value is fixed, not random, so it is not "95% likely" to be in one given interval. Confidence is a property of the procedure: repeat the sampling many times and about 95% of the intervals you build will contain the truth.
2 · You estimate a mean from a small sample and do not know the population sigma. Which interval, and why?
When sigma is unknown you estimate it, adding uncertainty. The t distribution's fatter tails widen the interval to account for that. As n grows, t converges to z and the two agree.
3 · Lumen's control converts at 3.2%; at n=5,000 its 95% CI is [2.71%, 3.69%]. The variant hopes for 3.6%. What does the interval tell you?
Because 3.6% falls inside the control's confidence interval at n=5,000, the data cannot distinguish it from 3.2%. That overlap is exactly why the experimentation course sizes tests at ~32,000 per arm to detect the effect.
What this session covers
This builder session covers ~80% of the computational confidence-interval content in the sources below, on the Lumen dataset. The interpretation - what "95%" does and does not mean - is the part most courses rush; we do not.
Builder Session 7 cheat sheet · pin this
stats.t.interval(0.95, df=n-1, loc=mean, scale=s/sqrt(n)).proportion_confint(k, n, method='normal' | 'wilson'). Small n*p → use Wilson.