"Conversion went from 3.2% to 3.6% - ship it!"
Lumen's new checkout page shows 3.6% conversion against the old page's 3.2%. That is a +0.4 point lift, a 12.5% relative gain - the room is already celebrating. But hold on: with a small sample, a jump like that happens by pure luck more often than you would guess. The entire discipline of hypothesis testing exists to answer one question honestly: is this difference real, or is it the kind of wobble you would see even if the two pages were identical? Get the logic right and you stop shipping noise as if it were signal.
Is it real, or is it luck? 8 min live
A statistical test runs a simple thought experiment. Start by assuming there is no real effect - the two checkout pages are actually identical. Then ask: if that were true, how surprising would our data be? If a 3.2%-to-3.6% gap would almost never happen by chance alone, we doubt the "no effect" assumption and call it real. If a gap that size shows up all the time by luck, we stay unconvinced. That surprise, put on a number, is the p-value.
LiveWhat a p-value is - and the two things it is not4 min▶
The p-value answers exactly one question: if there were no real effect, how often would we see data at least this extreme just by chance? Written precisely, it is P(data this extreme | no real effect is true). A small p-value (say under 0.05) means "this data would be surprising if nothing were going on," so we doubt the "nothing" story.
- It is NOT the probability the result is a fluke. The p-value already assumes no effect and measures the data's surprise - it does not tell you the odds your finding is fake.
- It is NOT the probability there is no effect. It is not P(no effect | data); it runs the other direction - P(data | no effect). Flipping those two is the single most common statistical error in business.
- Type I error (false alarm) - you flag an effect that is not there. The 0.05 threshold is your accepted false-alarm rate. Type II error (missed effect) - you miss a real one. Power is the chance of catching a real effect when it exists, and it is what sample size buys you.
Lumen's test reads 3.2% vs 3.6%. The team wants to declare victory. But at a modest sample the p-value comes back around 0.25 - meaning a gap this big would show up a quarter of the time even if the pages were identical. That is not a fluke-probability of 25%; it is "the data is not surprising under no-effect." Not significant. Shipping now would be flagging noise. The fix is not to argue - it is to gather enough data to give the test the power to see a real 0.4-point lift.
LiveSignificant is not the same as important3 min▶
Here is the trap on the other side. With a huge sample, almost any difference becomes "statistically significant" - even one far too small to matter. Significance says "this effect is probably real"; it says nothing about "this effect is big enough to care about."
- Tiny effect, huge n, significant, worthless. A +0.01 point lift on ten million sessions can be significant and still not pay for the engineering to ship it.
- Always pair the p-value with an effect size and a confidence interval. The effect size says how big; the interval says how sure. Significance alone is half an answer.
- The exec question: not just "is it real?" but "is it real and big enough to change what we do?" Both have to be yes.
The fallacy gauntlet 7 min live
Even with the test logic right, four classic traps ambush good decisions every week. This is the ★ section - the leader's field guide to the statistical fallacies that look most convincing precisely when they are most wrong. The nastiest is Simpson's paradox, where an aggregate trend flatly reverses inside every subgroup.
LiveThe four traps, and how to disarm each one4 min▶
- P-hacking / multiple comparisons. Test 20 things at a 5% threshold and about 1 will look "significant" by pure chance - even if nothing is real. Disarm it by pre-registering what you will test before you look, and by correcting the threshold when you run many tests (Bonferroni, FDR). A "winner" found after slicing the data 20 ways is usually the chance one.
- Correlation is not causation. Two things moving together does not mean one causes the other - a hidden third factor often drives both. Disarm it by asking "what else could explain this?" and, when it matters, running a controlled experiment (the whole point of the sequel course).
- Simpson's paradox. An aggregate trend can reverse inside every subgroup (the chart above). A Lumen channel can look better overall yet convert worse in cleanser, serum, and moisturizer once you split by category - because its traffic is concentrated in a naturally strong category. Disarm it by always asking "does this hold when we break it down?"
- Base-rate neglect (callback to Session 2). A test that is 99% accurate for a condition that hits 1 in 1,000 is wrong most times it fires - because the rare true cases are swamped by false alarms. Disarm it by always asking "how common is the thing we are testing for in the first place?"
A Lumen analyst reports that Channel Nine has the best overall conversion and pushes to move budget into it. A leader asks the Simpson's question: does it win within each product category? Split the data and it converts worse than the alternatives in every single category - it only looked good because it happened to send most of its traffic to sets, which convert well for everyone. The aggregate reversed the truth. One question saved a six-figure misallocation.
Self-studyWhy pre-registration is the cheapest insurance you can buy3 min read▶
The single habit that neutralizes most of the gauntlet is deciding what you will measure, and what counts as a win, before you see the data.
- It kills p-hacking - you cannot fish for a significant slice if you named your one metric in advance.
- It forces an honest effect size - committing to "we need at least a +0.3 point lift to bother" stops a trivial-but-significant result from being spun as a win.
- It makes the test sizeable - once you know the effect you care about, you can compute how much data you need. That computation is exactly where the experimentation course begins.
The same gap, significant or not - you decide with n ▶ interactive
This is Lumen's actual A/B result: control at 3.2%, variant at 3.6%. Drag the users-per-arm slider and watch the exact same 0.4-point gap flip from "not significant" to "significant" as the sample grows. The effect never changes - only how much data you gave the test to see it. That is precisely why the experimentation course computes the required sample size before launching, instead of eyeballing a lift after the fact.
Read a Lumen results deck like an exec ◐ 30 min total
Next time a team brings you a "the test worked, let us ship it" deck, run it through these five questions. If any answer is missing, the result is not ready to act on.
- 1 · Was there a control group? Compared to what? A number with nothing to compare against is not a result.
- 2 · Was the test powered and the sample size pre-committed? Did they decide how much data to collect in advance, or stop the moment it looked good (which manufactures false wins)?
- 3 · Do we have the p-value AND the effect size AND the confidence interval? All three. Significance without size is half an answer; size without an interval hides the uncertainty.
- 4 · What happened to the guardrails? Did the "winning" change quietly hurt refunds, support load, or margin while lifting the headline metric?
- 5 · How many things were tested? If they compared 20 segments and reported the one that won, that win is probably chance - ask for the correction.
Three questions before you go 🎯 ◐ 90 seconds
1 · A Lumen test comes back with p = 0.25 on the 3.2%-vs-3.6% gap. What does that number mean?
A p-value is P(data this extreme | no real effect). It is NOT the probability the result is a fluke (A) and NOT the probability there is no effect (C). At p = 0.25 the data is unsurprising under "no effect," so we do not call it real.
2 · A change is "statistically significant" at ten million sessions but lifts conversion by +0.01 points. The right response is...
Significance is not importance. At huge n almost anything is significant. Pair the p-value with effect size and CI, then decide if the effect is big enough to matter.
3 · A channel has the best conversion overall but converts worse inside every product category once you split. This is...
Simpson's paradox: an aggregate trend can reverse within every subgroup because of how the groups are weighted. The channel only looked good because its traffic sat in a naturally strong category. Always ask "does it hold when we break it down?"
What this session covers
This capstone leader session teaches the reading skills behind hypothesis testing and the fallacy gauntlet - the ~80% you need to judge whether a result is real and worth acting on. The hands-on computation (two-proportion z-tests, power analysis, corrections in statsmodels) lives in Builder Session 8; the sources below go deeper.