Stats, experiments and the product case

Say this: "A p-value is the probability of seeing data at least as extreme as what we observed, if there were really no effect and the model's assumptions held. A small one means the data are hard to square with no effect." That is the ASA's first principle in spoken form: p-values "can indicate how incompatible the data are with a specified statistical model".

Then add one of the other principles, unprompted. The strongest in an interview is principle 5: a p-value does not measure the size of an effect or the importance of a result. Follow it with what you would report instead: "so I would always report the effect size with a confidence interval, and ask whether that size matters for the product." Principle 3 is the next best: decisions should not be based only on whether a p-value passes a threshold. One extra principle shows you know the statement exists; reciting all six sounds memorised.

"It's the probability the hypothesis is true." The p-value is computed by assuming there is no effect, so it cannot also be the probability that there is one. The ASA's second principle says p-values do not measure the probability that the studied hypothesis is true.

"It's the probability the result was due to chance." This one sounds careful, which is why it survives. The same second principle rules it out: p-values do not measure the probability that the data were produced by random chance alone. The difference is the direction of the condition. The p-value is "probability of data this extreme, given no effect", not "probability of no effect, given these data". If an interviewer pushes, saying exactly that sentence about direction is the answer they are hoping to hear.

The ASA Statement on Statistical Significance and P-Values was released on 7 March 2016 (Wasserstein and Lazar, The American Statistician). Its six principles, paraphrased closely from the ASA's release:

1. P-values can indicate how incompatible the data are with a specified statistical model.
2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

Effect size is not significance. With enough users, a difference too small to matter will come out with a tiny p-value; with too few, a difference worth a lot can come out with a large one. That is principle 5 at work, and it is why every experiment readout in the build-along reports the size of the change and a decision rule, not a pass or fail. For the mathematics behind all of this, Learn Statistics with Phoebe ↗ owns the subject.

1. Unit of randomisation. Who gets assigned to a version: a user, an account, a session, a store. Choose the unit the change acts on, and say why. If one person can see both versions, say how you would stop that.

2. One primary metric. The single number that decides. Naming two primary metrics is naming none, because one of them will move by chance.

3. Guardrails. Two or three numbers that must not get meaningfully worse, even if the primary improves: support contacts, latency, cancellations, revenue proxies.

4. Sample size and duration, decided in advance. From a baseline rate, the smallest change worth acting on, and the error rates you accept. Round the duration up to whole weeks so weekdays and weekends are both in it.

5. When you will look. "I would read it once, at the end date, and only look before then to check the data are flowing and the split is even." That sentence is the one interviewers remember.

A standard p-value assumes the sample size was fixed before anyone looked. Check every day and stop the first time it dips under .05, and you have given chance fourteen tries to cross the line instead of one. Even with no real effect, a reading that wanders will cross it sometimes, and the stop-on-green rule turns that one crossing into a launch.

Johari, Pekelis and Walsh (2015, arXiv 1512.04922) put it bluntly in their abstract: standard p-values and confidence intervals are “wholly unreliable if users endogenously choose samples sizes by continuously monitoring their tests”. Their paper goes on to propose "always valid" inference built to survive continuous monitoring. In an interview you need the problem and one remedy: fix the end in advance, or, if the business genuinely needs to look early, use a method designed for sequential looks and say so. The details of those methods belong to Learn Statistics with Phoebe ↗.

Ren's constructed story S2, "The A/B test that was stopped early", runs 260 seconds in the bank, the longest of the eight. Its honest fits are experiments strong, pushing back strong, metric choice thin. Part 2 gives Ren the vocabulary to tell it in under two minutes and to make it serve the stats round as well as the behavioural one.

Situation and task: a product manager wanted to stop a test on day four because the dashboard showed a significant result. Action: Ren explained that the test had been sized for two weeks, that a reading checked daily will cross the line by chance some of the time, and proposed keeping the planned end date while watching guardrails daily. Result: at the end date the lift was much smaller than on day four, and the team decided on a smaller rollout. Every detail is constructed; the shape is what you copy. If an interviewer then asks "what would you do if the business could not wait?", Ren has the second half of the Johari answer ready.

Primary metric: the one number that, if it moves, means the change did what it was for. Tie it to the goal you clarified, not to the feature: a "skip a week" button is for keeping subscribers, so the primary is about retention, not about how many people click skip.

Guardrails: what must not get meaningfully worse. For a subscription product that usually means boxes or orders delivered, support contacts and complaints, plus a technical one such as page errors.

Counter-metric: the number that would expose the primary being gamed. If "retained" means "has not cancelled", a feature that lets people skip forever can raise retention while nobody receives anything. A counter-metric such as "retained subscribers who received at least one box in weeks 5 to 8" catches that. Naming a counter-metric before the interviewer finds the hole is the single most convincing move in a metric question.

Decompose: "Boxes per year is subscribers, times boxes each per week, times weeks." State assumptions: say each guess and where it comes from, even roughly: "two million households is a big city; I will guess one percent subscribe." Compute: write the multiplication down and do it slowly; 2,000,000 × 1% = 20,000, × 0.75 = 15,000, × 52 = 780,000. Sanity-check: turn the answer into a unit you can picture. 15,000 a week is about 2,100 a day, which is a busy warehouse, not a corner shop and not a continent. If the check fails, say which assumption you distrust and change it out loud.

Interviewers grade the structure and the check, not the closeness of your answer to a number they hold. No study was found on how estimation answers are scored; this is common practice, and it is also simply how a careful analyst works.

Some product cases are not "should we ship" but "this metric fell 10 percent last week, why?". The same discipline applies in a different order: confirm the metric's definition and the data pipeline first, then split by time, platform, region and user type to find where the drop lives, then form hypotheses for that slice only. Saying "first I would check it is not a logging change" earns credit before any analysis. Product analytics as a subject, with funnels, cohorts and metric trees, belongs to Learn Product Analytics with Phoebe ↗.