A significant result can still be a broken one
The green "significant" badge only tells you the gap is unlikely to be pure luck. It says nothing about whether the test was built correctly in the first place. A test can be assigned unfairly, validated never, split 55/45 when it should be 50/50, flattered by the novelty of a new design, contaminated by users talking to each other, or winning on conversion while silently breaking refunds. None of those show up as "not significant" - they show up as a confident, wrong recommendation. Your defence is a checklist you run before you let the lift number into the conversation.
Six questions to ask before you believe a test 10 min live
Run these in order. The early ones are cheap and catch the most damage - a test that fails randomization or SRM is not worth reading further. Each is framed the way you would actually say it in a results review, followed by why it matters and the Lumen story that makes it concrete.
Live1 · "Was assignment random, and at the right unit?"2 min▶
Randomization is the whole reason an A/B test can claim causation - it makes the two groups comparable on everything, including things you never measured. If people chose their own group, or were split by something non-random (signup date, region, device), the groups are no longer twins and any gap is contaminated. The unit matters too: if one shopper can land in both groups across visits, the comparison leaks.
Lumen once "tested" a new checkout by giving it to everyone on mobile and keeping desktop on the old flow. Mobile shoppers convert differently for a dozen reasons, so the "lift" was mostly a device difference, not the checkout. The fix was to randomize each visitor by a coin flip within every device, so the only systematic difference between the groups was the checkout itself.
Live2 · "Did we validate the pipeline with an A/A test?"2 min▶
An A/A test shows both groups the same thing and checks that the measurement machinery reports "no difference", as it should. It is a smoke test for the plumbing: tracking, bucketing, and the stats. If an A/A test finds a "winner" between two identical experiences, something is broken - and you would never have known from an A/B result alone.
Before trusting its experimentation platform, Lumen ran an A/A on the product page. A clean A/A returns roughly uniform p-values and flags "significant" about 5% of the time by chance - exactly as expected. Their first attempt instead showed a consistent 0.3-point gap between identical pages, which traced to a logging bug that double-counted one group. They fixed the pipeline before running a single real test. An A/A that "wins" is a gift - it tells you the numbers cannot be trusted yet.
Live3 · "Were the groups the size we intended?" (Sample Ratio Mismatch)2 min▶
You planned a 50/50 split. If the data shows 52/48, or worse, something decided who landed where - a redirect that dropped slow connections, a bot filter that hit one arm, a bucketing bug. This is Sample Ratio Mismatch (SRM), and it is a red flag that invalidates the test: if the split is not what you intended, the groups are no longer comparable, and the lift is untrustworthy no matter how significant it looks.
- SRM is checked with a quick statistical test, and the alarm threshold is strict - roughly 1 in 1,000, not the usual 1 in 20 - because even a small imbalance signals a real problem.
- The right response to SRM is not to interpret the result. It is to stop, find the cause, and rerun. A test with SRM is a broken test, full stop.
Lumen's hero-first test was planned as 50/50 across roughly 64,000 users - about 32,000 per arm. The deck arrived showing 33,700 in control and 30,300 in variant. That lopsided split was SRM: a caching rule had quietly sent some variant traffic back to the old page. The "+0.5 point win" was meaningless because the two groups were no longer built the same way. They stopped, fixed the cache, and reran - and the honest result was smaller.
Live4 · "Is the win just novelty (or the reverse, primacy)?"2 min▶
People react to new, not just to better. A fresh design can get a bump purely because it is unfamiliar and grabs attention - the novelty effect - which fades once it becomes the norm. The opposite, the primacy effect, is when loyal users are briefly annoyed by change and the new version looks worse than it truly is. Both are temporary reactions to change itself, not to the change's real value.
A bold new Lumen homepage banner won big in week one. But when the team split the results by new versus returning visitors, the entire lift came from returning shoppers clicking the shiny new thing - and it decayed sharply by week two. New visitors, who had no "old" to compare against, showed no lift at all. The durable effect was near zero. The tell: a win concentrated in returning users that shrinks over time is usually novelty, not value.
Live5 · "Could the treatment have leaked into the control?" (interference)2 min▶
A clean test assumes each group's experience is independent - what happens to the variant does not touch the control. That assumption (statisticians call it SUTVA) breaks when users interact, share, or compete for the same finite resource. When treatment "leaks" into control, both groups move together and the true effect is masked or faked.
- Marketplaces and shared inventory - if the variant makes buyers grab limited stock faster, the control group sees fewer items available. You changed the control by testing.
- Referrals and social sharing - a variant user sends a discount link to a control-group friend. The treatment jumped the fence.
Lumen tested an aggressive "refer a friend" reward on half its users. Those users shared codes with friends who happened to be in the control group, lifting control's conversion too. The measured gap between groups looked small, badly understating the program's real impact. When the units can talk to each other, splitting individuals is the wrong design - which is exactly why Session 4 reaches for geo tests, where whole regions are the unit.
Live6 · "Did anything that must-not-break get worse?" (guardrail metrics)2 min▶
A test optimises one headline metric, but the business runs on many. Guardrail metrics are the numbers that must not degrade even if the headline wins - the vetoes. A conversion lift that quietly doubles refunds, slows the page, or tanks add-to-cart is not a win; it is a problem with good marketing.
- For Lumen the guardrails are page-load latency, refund rate, and add-to-cart rate. A win on conversion that breaks any of these is not shipped.
- Guardrails are agreed before the test, so nobody can quietly drop an inconvenient one after the results are in.
What Sample Ratio Mismatch looks like 4 min live
SRM is the checklist item with the least judgement and the most force: either the groups are the intended size or they are not, and if they are not, you stop. Here is the picture - what you planned versus what the broken test delivered.
Six gates before "believe the result" 4 min live
Think of the checklist as a ladder the test must climb before its lift number earns your trust. Each rung is one question. Fail any rung and you do not proceed to the next - you send the test back. Only a test that clears all six has earned a place in the decision.
Stopping when it "looks good" fakes a winner try it
The checklist assumes the test ran to its planned finish line. This tool shows why that matters. Both versions here are truly identical - there is no real difference. The test is designed to run to ~32,000 per arm. Watch how often peeking early and stopping at the first "significant" moment declares a false winner, far above the honest 5%. Nobody cheated; they just looked too soon. This is why a trustworthy result names its sample size before the test starts, and sticks to it.
This week ◐ 30 min total
- Run the six-question gate on a real experiment in your org. For each question, mark pass, fail, or "the deck doesn't say". Count how many are "doesn't say" - that gap is usually where trust quietly leaks.
- Pick your guardrails. For one upcoming test, write down the two or three metrics that must NOT get worse even if the headline wins. Get them agreed before the test runs.
- Find a novelty suspect. Look for a past "win" that was strongest in week one among returning users. Ask whether anyone checked if it lasted.
- Play with the peeking tool above. Set the peeks high and watch the false-win rate climb even though nothing is truly different. Bring the number you see to Session 4.
Three questions before you go 🎯 ◐ 90 seconds
1 · A test planned as 50/50 arrives showing 33,700 in control and 30,300 in variant. What should you do?
SRM means something decided who landed where, so the groups are no longer twins. The alarm threshold is strict (~1 in 1,000). Do not interpret the lift - debug the cause and rerun. Dropping users does not fix the underlying bias.
2 · A new banner wins big in week one, entirely among returning visitors, and the lift fades by week two. The most likely explanation is...
A win concentrated in returning users that shrinks over time is the classic novelty signature. New visitors, with no "old" to compare, showed no lift - so the durable effect was near zero.
3 · An A/A test (both groups shown the same thing) reports a "significant" winner. What does that tell you?
Identical experiences should show no real difference (a clean A/A flags "significant" only ~5% of the time by chance). A consistent "winner" between two identical groups points to a plumbing bug - tracking, bucketing, or the stats - that must be fixed first.
What this session covers
This leader session distills the validity threats that separate a trustworthy test from a broken one - the conceptual ~80%. The statistical mechanics (the SRM chi-square, the randomization code) live in the builder track and the sources below.