One sample is a guess - the shape of many samples is a law
Take 100 Lumen orders and average them: you get a number near $74, but not exactly. Take another 100, another number. Do that thousands of times and something remarkable happens - those averages pile up into a clean, symmetric bell, even though the raw order values are stubbornly right-skewed. That bell has a known width, and knowing its width is what lets you say "the true mean is $74, give or take." Today you build that bell with your own hands.
SE = sigma / sqrt(n) verified against your own simulation, and the Central Limit Theorem shown - not asserted - on Lumen's skewed money data. Plus the bootstrap: a standard error and a confidence interval for the mean with no formula at all, just resampling.
Sampling distributions, standard error, and the CLT 6 min live
Three ideas stack on top of each other. Get them in order and the rest of the course is downhill.
LiveThe three ideas, in order3 min▶
- Sampling distribution of a statistic. Any statistic you compute (a mean, a median, a proportion) is itself random - draw a new sample and it changes. The distribution of that statistic across many samples is the sampling distribution. It is the thing statistics actually reasons about.
- Standard error. The standard deviation of that sampling distribution has a name: the standard error. For a mean it is
SE = sigma / sqrt(n). More data (bigger n) means a narrower sampling distribution, but only as fast as the square root - quadruple n to halve the SE. - The Central Limit Theorem. As n grows, the sampling distribution of the sample mean approaches a normal distribution, regardless of the shape of the population. Skewed, bimodal, lumpy - it does not matter. The mean goes normal. That universality is why the normal curve is everywhere in statistics.
Watch the bell emerge from skewed money ★ 22 min · everyone builds
Reload the Lumen orders you built in b1, then simulate sampling itself. We draw 2,000 samples of size n, take each sample's mean, and histogram those means. Bump n up and the histogram turns into a bell - even though the raw data never does.
Reload the data (seed 42). Same file you saved in b1 - the numbers reconcile.
import numpy as np, pandas as pd
df = pd.read_parquet("lumen_orders.parquet") # built in b1
ov = df["order_value"].to_numpy()
print("N :", ov.size) # 50000
print("mean :", round(ov.mean(), 2)) # ~ 74 (the population mean)
print("std :", round(ov.std(), 2)) # ~ 38 (the population sigma)
print("skew :", round(pd.Series(ov).skew(), 2)) # ~ +0.9 still skewed
The money demo. Draw 2,000 sample means for n = 2, 10, 50 and watch the bell arrive.
rng = np.random.default_rng(42)
REPS = 2000
def sample_means(data, n, reps, rng):
# reps samples of size n, return the mean of each
idx = rng.integers(0, data.size, size=(reps, n))
return data[idx].mean(axis=1)
for n in (2, 10, 50):
means = sample_means(ov, n, REPS, rng)
print(f"n={n:>2} mean-of-means={means.mean():6.2f} "
f"std-of-means={means.std():5.2f} "
f"skew={pd.Series(means).skew():+.2f}")
# n= 2 mean-of-means= 74.0x std-of-means=26.9x skew=+0.6x (still lumpy)
# n=10 mean-of-means= 74.0x std-of-means=12.0x skew=+0.2x (rounding out)
# n=50 mean-of-means= 74.0x std-of-means= 5.3x skew=+0.0x (normal)
The mean of the sample means stays glued to the population mean (~$74). What changes is the spread - it shrinks - and the skew - it melts toward zero. That melting is the CLT happening in front of you.
Verify the standard error. The spread you just measured should match sigma / sqrt(n).
sigma = ov.std() # population SD ~ 38
for n in (2, 10, 50):
means = sample_means(ov, n, REPS, rng)
observed = means.std() # spread of the sampling distribution
formula = sigma / np.sqrt(n) # the SE formula
print(f"n={n:>2} observed={observed:5.2f} "
f"formula={formula:5.2f} ratio={observed/formula:4.2f}")
# ratios land near 1.00 - the formula and the simulation agree.
# n=50: 38 / sqrt(50) ~= 5.37, and your histogram width ~= 5.3.
Bootstrap - a standard error with no formula. Resample the data with replacement, recompute the mean each time, and read the spread straight off the resamples.
rng = np.random.default_rng(42)
B = 2000
n = ov.size
boot_means = np.empty(B)
for b in range(B):
resample = rng.choice(ov, size=n, replace=True) # same size, WITH replacement
boot_means[b] = resample.mean()
se_boot = boot_means.std()
ci_lo, ci_hi = np.percentile(boot_means, [2.5, 97.5]) # 95% percentile CI
print("bootstrap SE :", round(se_boot, 3)) # ~ 0.17
print("formula SE :", round(ov.std()/np.sqrt(n), 3)) # ~ 0.17 they agree
print("95% CI for mean:", [round(ci_lo, 2), round(ci_hi, 2)]) # ~ [73.7, 74.3]
The bootstrap is the practitioner's escape hatch. When you need a standard error for something with no clean formula - a median, a 90th percentile, a conversion-weighted average order value - you resample and measure the spread directly. No distributional assumption, no lookup table. It is the same engine as the CLT, run by brute force, and it is why every confidence interval and A/B readout in the rest of this course rests on the ideas you built today.
Push n up and watch the bell tighten interactive
The code you just ran, live in the page. Drag the sample size up and watch the sample-mean distribution tighten and normalize on Lumen's skewed order values - exactly the money demo above, now under your fingers.
This week ◐ 40 min total
- Sample a proportion, not a mean. Build the
convertedflag at ~3.2% (rng.random(N) < 0.032), then draw 2,000 samples of size n=1,000 and histogram the sample conversion rates. Confirm the spread matchessqrt(.032*.968/1000)~= 0.56pp. - Break the CLT (almost). Rerun the money demo at n=2 and n=5. How large does n have to get before the sample-mean histogram looks convincingly bell-shaped on this skewed data? Write one sentence.
- Bootstrap a median. Repeat the bootstrap but take
np.medianof each resample. You now have a standard error for the median - something with no simple formula. - Optional: overlay
scipy.stats.norm.pdf(with mean = ov.mean(), SD = sigma/sqrt(n)) on your n=50 histogram. The fit is the CLT, drawn.
Three questions before you go 🎯 ◐ 90 seconds
1 · Lumen order values are strongly right-skewed. What does the Central Limit Theorem promise as sample size n grows?
The CLT is about the sampling distribution of the mean, not the raw data. Order values stay skewed forever; the average of a sample of them goes normal as n grows. That is the single most misread sentence in statistics.
2 · Your sample-mean histogram at n=10 has spread ~12. You move to n=40. Roughly what spread do you expect?
SE = sigma / sqrt(n). Going from n=10 to n=40 multiplies n by 4, which multiplies sqrt(n) by 2, so the standard error is halved: 12 becomes about 6. Quadruple the data to halve the wobble.
3 · What is the defining feature of a bootstrap resample when estimating a standard error?
Bootstrapping resamples the observed data itself at the original size, with replacement, so some points repeat and some drop out. The spread of the statistic across those resamples is the standard error - no formula and no distributional assumption required.
What this session covers
This builder session covers ~80% of the computational sampling-and-CLT content in the sources below, on the Lumen dataset. The formal proof of the theorem is theory you can read; the simulation is the intuition you keep.
Builder Session 5 cheat sheet · pin this
SE = sigma / sqrt(n). Quadruple n to halve the SE. The square-root wall.rng.choice(data, size=len(data), replace=True) in a loop. SE and CI with no formula.