Good tests are a habit, not an event
A single well-run experiment is worth something. An organization that runs a hundred a year, each one honest, each one feeding the next decision, is worth a great deal more - and that is a leadership artifact, not a statistical one. The statistics are settled by Sessions 1 to 5. What is left is the part only you can build: agreeing on the success metric up front, protecting the numbers that must not break, deciding in advance what a result will mean, and building a team that treats a failed test as cheap learning rather than a black mark. This capstone is about that system.
Decide what winning means before the test runs 9 min live
The most common way a test goes wrong has nothing to do with statistics. It is that nobody agreed, before the test, what result would count as a win - so afterwards everyone finds the metric that flatters their side. The fix is to name one Overall Evaluation Criterion (OEC) up front: the single number this test is judged on. Then name the guardrails - the metrics that must not get worse even if the OEC wins. And then pre-commit to what each outcome triggers: ship, iterate, or kill.
LiveThe OEC and why it must be singular3 min▶
The Overall Evaluation Criterion is the one metric a test is graded on, agreed before launch. Its whole power is that it is singular and pre-committed - it takes away the after-the-fact temptation to cherry-pick whichever number moved.
- One metric, chosen up front - for Lumen's page test, checkout conversion. Not "conversion, or revenue, or time-on-page, whichever looks good later".
- It should capture long-term value, not a vanity spike - clicks can rise while revenue falls. A good OEC is the number you would actually stake the business on.
- Agreed by the people who will act on it - so the result is binding, not debatable.
Lumen's hero-first page test names checkout conversion as its OEC before launch - target lift from 3.2% to 3.6%. When the test comes in at exactly that, there is no argument about whether it "worked", because the bar was set in advance. That is the OEC doing its job.
LiveGuardrails and the pre-committed decision4 min▶
Guardrail metrics are the things that must not get worse even when the OEC wins. They stop a team from buying a headline number at the cost of something quietly expensive.
- Lumen's guardrails - page-load latency, refund rate, add-to-cart rate, unsubscribe rate. A conversion win that slows the page by 180ms is not shipped until the latency is fixed.
- The decision, pre-committed - before the test, agree what each outcome triggers: ship (OEC wins, guardrails hold), iterate (promising but not conclusive, or a guardrail wobbled), kill (flat or negative). Deciding in advance removes the emotion afterwards.
- Absence of evidence is not evidence of absence - a "not significant" result from Session 2 usually means iterate or re-power, not "the idea is dead".
Velocity, the ROI of failure, and what rots a culture 7 min live
Most experiments fail - the change does not move the OEC, or it moves it the wrong way. In a healthy culture that is the point, not a problem. Each cheap failed test buys a piece of knowledge and steers the next bet. The return on experimentation comes from volume plus honesty: run many tests, learn from all of them, ship the few real winners. The leader's job is to make failing safe, keep the loop turning, and stamp out the behaviours that quietly turn the whole system into theatre.
LiveThe ROI of experimentation is mostly learning3 min▶
If you only counted the winning tests, experimentation would look inefficient - most tests do not win. That framing misses the point. The value is in the portfolio: cheap failures that stop you shipping expensive mistakes, plus the occasional real winner that pays for all of them.
- Most tests fail, by design - if nearly all your tests win, you are testing only safe bets and learning little. A healthy program has plenty of flat and negative results.
- Failures are cheap insurance - a $32,000-per-arm test that kills a bad idea saved you from rolling that idea out to everyone.
- Reward the test, not the outcome - a well-designed test that returned a clean "no" is a success. Celebrating only positive results teaches teams to game the metric.
Lumen's growth team runs about 40 tests a year. Roughly 8 ship, a handful iterate, the rest are killed. The CMO reports the program by learnings banked and mistakes avoided, not by win rate - which is exactly why the team keeps proposing bold, risky tests instead of safe ones.
Self-studyThe org anti-patterns that rot experimentation3 min read▶
Every failure mode here is cultural, not statistical - which means only a leader can fix it.
- The HiPPO override - the Highest-Paid Person's Opinion overturns a clean result because they do not like it. Kills the incentive to test honestly. If you commission an experiment, you agree to be bound by it.
- P-hacking to a win - slicing the data until some segment looks significant, then reporting that. The peeking trap from Session 1, dressed up. Pre-commit the OEC and the analysis.
- Celebrating only positive results - if wins get applause and honest "no" results get silence, teams learn to manufacture wins. Reward the quality of the test.
- No pre-committed decision - running a test with no agreement on what the outcome triggers, so the result gets argued away.
Commission Lumen's experimentation roadmap ◐ 30 min
This is the whole leader track in one exercise. Lumen's CMO must reallocate a $4M media budget defensibly across 9 channels. Play the leader who commissions the evidence.
- Pick 3 decisions from the $4M budget question. For example: (a) does paid search deserve its 24% share, or is it mostly demand Lumen already has; (b) should the CTV line at 8% grow; (c) is the new hero-first product page worth shipping site-wide.
- Assign each to ship / pilot / experiment using Session 1's stakes-and-uncertainty grid. High stakes and high uncertainty earns a full experiment; high stakes but confident earns a pilot or staged rollout; low stakes and reversible just ships.
- Match each experiment to its tool. Paid search incrementality wants a holdout (Session 4). CTV wants a geo test across the 9 regions (Session 4). The product page wants a user-level A/B at ~32,000 per arm (Sessions 2 to 3). If a change already rolled out unevenly, reach for a quasi-experiment (Session 5).
- Name the OEC and guardrails for the biggest one. For the product-page test: OEC = checkout conversion (target 3.2% to 3.6%); guardrails = page-load latency, refund rate, add-to-cart rate, unsubscribe rate. State the pre-committed decision: ship if the OEC wins and guardrails hold, iterate if inconclusive, kill if flat or negative.
- Write the one-paragraph commission. Hand it to your team as if it were real: what you are testing, why it earns an experiment, the tool, the OEC and guardrails, and what each result will make you do. That paragraph is the artifact of an experimentation leader.
Three questions before you go 🎯 ◐ 90 seconds
1 · Why name the OEC before the test runs rather than after?
The OEC's power is that it is singular and agreed up front. Choosing the metric after seeing the data is how p-hacking and motivated reasoning creep in.
2 · Lumen's page test lifts conversion to target, but page-load latency gets 180ms worse. What should happen?
Guardrails are the metrics that must not get worse even when the OEC wins. A conversion win bought with a latency regression is not a win until the guardrail is restored.
3 · Most of Lumen's tests come back flat or negative. What does that tell a good leader?
Most experiments fail by design, and that is the point. The ROI comes from cheap failures that prevent expensive mistakes plus the occasional real winner. Reward good tests, not just wins.
What this session covers
This capstone distills the culture chapters of the experimentation literature - the part the technical courses assume a leader already owns. It covers ~80% of the conceptual content; the rest lives in the sources below.