When the question is counts, or more than two groups
A t-test is for two means of continuous data. But plenty of real questions are not that shape. "Did our 50/50 experiment actually split 50/50?" is about counts. "Is conversion related to channel?" is about a table of counts. "Do our four product lines have different average order values?" is three-plus groups at once. Chi-square handles the first two; ANOVA handles the third - and doing it right means NOT firing off a pile of t-tests.
scipy.stats.chisquare, chi2_contingency), one-way ANOVA (f_oneway), and the nonparametric fallbacks (mannwhitneyu, kruskal) for when normality or equal-variance fails. Plus the discipline behind them: chi-square eats counts not proportions, and ANOVA tells you something differs, not which.
Chi-square on counts, ANOVA on group means 13 min live
Two ideas, one figure. Chi-square compares what you observed to what you expected. ANOVA asks whether the gaps between group means are big relative to the noise within the groups.
LiveChi-square: goodness-of-fit and independence5 min▶
- Goodness-of-fit: one categorical variable against an expected split. "Did 20,000 assignments land 50/50?" Compares observed counts to expected counts, one row per category.
- Independence: two categorical variables in a contingency table. "Is conversion related to channel?" H0 is that the two are independent - the row and column proportions do not interact.
- The statistic: chi-sq = sum of (observed - expected)² / expected across cells. Bigger = further from what the null predicts; scipy turns it into a p-value.
LiveANOVA: one F for many groups, and its assumptions5 min▶
- Why not many t-tests: four groups make six pairwise t-tests. At alpha 0.05 each, the chance of at least one false positive climbs past 25%. ANOVA asks the one question - "are they all equal?" - at a single, controlled alpha.
- The F-statistic: between-group variance divided by within-group variance. If the group means are spread far apart relative to the noise inside each group, F is large and p is small.
- Assumptions: roughly normal residuals and roughly equal variances across groups. When those fail (skewed money data, wildly different spreads), switch to a nonparametric test.
- What it does NOT tell you: a significant ANOVA says "at least one group differs", not which one. That needs a post-hoc test (Tukey HSD) afterward.
mannwhitneyu is the rank-based stand-in for a two-sample t-test; kruskal (Kruskal-Wallis) is the stand-in for one-way ANOVA. They compare distributions by rank, so heavy skew and unequal variance stop being a problem.
SRM check, an independence table, one-way ANOVA, and a rank test ★ 24 min · everyone builds
Reload the Lumen dataset from b1 (with the converted flag). We add a channel column - the 9-value enum from the touchpoints table - so we can build a real contingency table.
Chi-square goodness-of-fit as a sample-ratio-mismatch check. A "50/50" A/B assignment of 20,000 users came back 10,420 / 9,580. Did randomization break?
from scipy import stats
observed = [10_420, 9_580] # counts landing in variant / control
expected = [10_000, 10_000] # a true 50/50 split of 20,000
chi2, p = stats.chisquare(f_obs=observed, f_exp=expected)
print(f"chi2 = {chi2:.1f}, p = {p:.6f}") # p ~ 0.0000 -> SRM, STOP
# A 4.2%/-4.2% imbalance across 20,000 is far too big to be luck.
# Do NOT read the conversion result off a broken split - fix the splitter first.
This is the exact guardrail the experimentation course (b3) runs before it trusts any A/B result. If the assignment ratio itself fails a chi-square test, something upstream is dropping or double-counting users, and the shiny conversion lift downstream is not to be believed. One chisquare call, and you either proceed or you go debug the pipeline.
Chi-square test of independence. Is whether a session converts independent of which channel it came from? Build the channel-by-converted contingency table.
import numpy as np, pandas as pd
df = pd.read_parquet("lumen_orders.parquet")
rng = np.random.default_rng(42)
channels = ["organic","paid_search","email","affiliate",
"social","display","referral","direct","sms"]
df["channel"] = rng.choice(channels, len(df))
df["converted"] = rng.random(len(df)) < 0.032 # ~3.2% overall
table = pd.crosstab(df["channel"], df["converted"]) # COUNTS, not rates
chi2, p, dof, expected = stats.chi2_contingency(table)
print(table)
print(f"chi2 = {chi2:.2f}, dof = {dof}, p = {p:.3f}")
# assigned at random here -> p large -> no evidence conversion depends on channel
One-way ANOVA. Do the four product categories share one mean order value? One F, not six t-tests.
groups = [df.loc[df.product_category == c, "order_value"]
for c in ["cleanser", "serum", "moisturizer", "set"]]
f, p = stats.f_oneway(*groups)
print(f"F = {f:.2f}, p = {p:.3f}")
# In our b1 sim order_value does not depend on category -> p large.
# If p WERE small, ANOVA says "some group differs" - then run Tukey HSD:
# from statsmodels.stats.multicomp import pairwise_tukeyhsd
# pairwise_tukeyhsd(df.order_value, df.product_category) # which pair(s)?
Nonparametric fallback. Order value is right-skewed, so a t-test's normality assumption is shaky. Compare cleanser vs serum by rank instead.
cleanser = df.loc[df.product_category == "cleanser", "order_value"]
serum = df.loc[df.product_category == "serum", "order_value"]
u, p = stats.mannwhitneyu(cleanser, serum, alternative="two-sided")
print(f"Mann-Whitney U = {u:.0f}, p = {p:.3f}")
# ranks, not raw values -> skew and unequal variance stop mattering.
# The 3+ group version is Kruskal-Wallis:
k, pk = stats.kruskal(*groups)
print(f"Kruskal-Wallis H = {k:.2f}, p = {pk:.3f}")
This week ◐ 40 min total
- Break the split on purpose. Change the SRM counts to 10,050 / 9,950 and re-run
chisquare. At what imbalance does p cross 0.05? That is your SRM sensitivity. - Make a real dependence. Give one channel a higher conversion rate when you build
converted, then re-runchi2_contingency. Confirm p drops and the observed table diverges fromexpected. - Force an ANOVA signal. Add $15 to
setorder values, re-runf_oneway, and if it is significant runpairwise_tukeyhsdto see which pair drives it. - Optional: compare the t-test and Mann-Whitney p-values on the same two groups. When do they agree, and when does skew pull them apart?
Three questions before you go 🎯 ◐ 90 seconds
1 · You want to chi-square test whether a 50/50 split held. What do you feed the test?
Chi-square is defined on counts, never proportions. The same percentage gap means something completely different at n=200 versus n=20,000, and only the counts carry that information.
2 · Why run one ANOVA on four groups instead of six pairwise t-tests?
Every extra test is another roll at a false positive. Six independent tests at alpha 0.05 push the family-wise error past 25%. ANOVA asks the single question at one controlled alpha.
3 · A one-way ANOVA on four categories comes back p = 0.002. What can you conclude?
ANOVA is an omnibus test: it flags that the groups are not all equal, without naming the culprit. A post-hoc test like Tukey HSD tells you which specific pair(s) differ.
What this session covers
This builder session covers ~80% of the introductory chi-square and ANOVA content in the sources below, on the Lumen dataset. Post-hoc tests and full experiment design live in the experimentation course; correlation and regression close this track in b10.
Builder Session 9 cheat sheet · pin this
stats.chisquare(f_obs, f_exp). One category vs an expected split. The SRM check.stats.chi2_contingency(crosstab). Two categoricals. H0: they are independent.stats.f_oneway(g1, g2, g3, ...). F = between-group var / within-group var.