Hypothesis testing, in scipy and statsmodels

Lumen's growth team ran a new checkout button and saw 3.6% convert versus the old 3.2%. Is that a real lift, or the kind of wobble you get from luck alone? A hypothesis test is the disciplined answer: assume nothing changed, ask how surprising the data would be under that assumption, and decide. The trap is reading the output wrong - so today we get the p-value definition exactly right and pair every verdict with an effect size and a confidence interval.

Every test is the same six moving parts. Learn them once here and the code in Part 2 is just plumbing.

Reload the Lumen dataset you built in b1 (add the converted flag from the b1 homework if you have not). Each test is a few lines - the skill is stating the null out loud before you read the number, and reading the number honestly after.

import numpy as np, pandas as pd
from scipy import stats

df = pd.read_parquet("lumen_orders.parquet")   # from b1
ov = df["order_value"]

t, p = stats.ttest_1samp(ov, popmean=70)
print(f"mean = {ov.mean():.2f}")        # ~ 74
print(f"t = {t:.2f}, p = {p:.2e}")      # large t, tiny p -> reject H0
# effect size (Cohen's d) - is the gap PRACTICALLY meaningful?
d = (ov.mean() - 70) / ov.std(ddof=1)
print(f"Cohen's d = {d:.3f}")           # small-ish: significant != big

cleanser = df.loc[df.product_category == "cleanser", "order_value"]
serum    = df.loc[df.product_category == "serum",    "order_value"]

# Welch's t-test does NOT assume equal variances - the safer default
t, p = stats.ttest_ind(cleanser, serum, equal_var=False)
print(f"cleanser mean {cleanser.mean():.2f} vs serum {serum.mean():.2f}")
print(f"t = {t:.2f}, p = {p:.3f}")      # p large -> fail to reject
# failing to reject is NOT proof they are equal - it is "no evidence of a gap"

from statsmodels.stats.proportion import proportions_ztest

def ab(n_per_arm, p_ctrl=0.032, p_var=0.036):
    conv  = np.array([round(p_var * n_per_arm), round(p_ctrl * n_per_arm)])
    nobs  = np.array([n_per_arm, n_per_arm])
    z, p = proportions_ztest(conv, nobs)   # variant first
    return z, p

z1, p1 = ab(5_000)
z2, p2 = ab(32_000)
print(f"n=5,000/arm : z = {z1:.2f}, p = {p1:.3f}")   # z~1.16, p~0.25  NOT significant
print(f"n=32,000/arm: z = {z2:.2f}, p = {p2:.3f}")   # z~2.8,  p~0.005 significant

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

h = proportion_effectsize(0.036, 0.032)   # Cohen's h ~ 0.022 (tiny effect)
n = NormalIndPower().solve_power(
        effect_size=h, alpha=0.05, power=0.80, alternative="two-sided")
print(f"Cohen's h = {h:.4f}")
print(f"n per arm  = {n:,.0f}")   # ~ 32,000 - matches the z~2.8 crossover above

Here is Lumen's A/B result as a live tool. Drag users per arm and watch the 3.2%-vs-3.6% gap cross from "not significant" to "significant" - the CI on the lift closing in on zero and then finally excluding it. Same effect the whole time; only your certainty changes.

• Find the crossover yourself. Loop ab(n) over n from 5,000 to 40,000 in steps of 2,500 and print the first n where p drops below 0.05. Compare it to the power calculation's answer.
• One-tailed vs two-tailed. Re-run the one-sample t-test with alternative="greater". How does p change, and why is committing to a direction beforehand the only thing that makes it honest?
• Effect size discipline. Invent a scenario where a test is significant (p < 0.05) but the effect is too small to act on. Write one sentence a stakeholder would understand.
• Optional: simulate the Type I rate - run ttest_ind on two samples drawn from the SAME distribution 1,000 times and confirm about 5% of them come back p < 0.05 anyway.

This builder session covers ~80% of the introductory hypothesis-testing content in the sources below, on the Lumen dataset. Chi-square and ANOVA get their own session (b9); correlation and regression close the track in b10.