Relationships - and the ways they lie to you
"Prior spend predicts this order." "This channel converts best." Correlation and regression are how you make claims like that with numbers behind them - and they are also where confident people go badly wrong. A single correlation coefficient can hide a curve. Twenty random features will hand you a "significant" one by pure luck. A channel that wins overall can lose inside every single segment. This capstone teaches the tools and the traps together, because in practice they arrive together.
statsmodels (reading R-squared, coefficients, and residuals), and the three fallacy defenses: Anscombe's warning that r measures linear only, multiple-comparison correction (Bonferroni + FDR), and a worked Simpson's paradox on Lumen. The r ~ 0.6 you measure here is the exact CUPED covariate the experimentation course uses to shrink variance.
Pearson vs Spearman, and reading a regression 13 min live
Correlation is one number for "do these move together". Regression is the line (or plane) that quantifies how. The danger is that the single number can look identical across wildly different pictures - which is what the figure shows.
LivePearson, Spearman, and what r cannot see4 min▶
- Pearson r: measures linear association only, from -1 to +1. A strong curve can sit at r ~ 0 - the relationship is real, just not straight.
- Spearman: Pearson on the ranks. Captures any monotonic relationship (as one goes up the other consistently goes up), robust to outliers and nonlinearity.
- r-squared: the square of r for a simple regression - the share of variance in the outcome the predictor explains. r ~ 0.6 means r-squared ~ 0.36, so ~36% explained.
- The cardinal rule: correlation is not causation. A tight r can come from a lurking third variable, reverse causation, or pure coincidence.
LiveReading an OLS regression: coefficients, R-squared, residuals5 min▶
- Coefficients: in
order_value ~ prior_30d_spend, the slope is "dollars of order value per extra dollar of prior spend, holding other terms fixed". The intercept is the fitted value when predictors are zero. - R-squared: share of order-value variance the model explains. Adjusted R-squared penalizes adding useless predictors.
- Residuals: observed minus predicted. Plot them against the fitted values - a shapeless cloud is healthy; a funnel or a curve means the model is missing structure.
- Multiple regression: add categorical terms with
C(product_category)and each coefficient becomes a comparison to the baseline category, adjusted for prior spend.
Correlate, regress, then walk into every trap on purpose ★ 24 min · everyone builds
Reload the Lumen dataset from b1. First measure and model the prior-spend relationship, then run the three fallacy demos that separate a careful analyst from a confident wrong one.
Pearson + Spearman. How does prior-30-day spend relate to this order's value? This r is the variance-reduction covariate the experimentation course reuses.
import numpy as np, pandas as pd
from scipy import stats
df = pd.read_parquet("lumen_orders.parquet") # from b1
r_p, p_p = stats.pearsonr(df.prior_30d_spend, df.order_value)
r_s, p_s = stats.spearmanr(df.prior_30d_spend, df.order_value)
print(f"Pearson r = {r_p:.2f} (r^2 = {r_p**2:.2f})") # ~ 0.60, r^2 ~ 0.36
print(f"Spearman r = {r_s:.2f}") # similar - monotonic
# r^2 ~ 0.36 means prior spend explains ~36% of order-value variance.
# That is exactly the CUPED covariate that shrinks A/B variance in experimentation b7.
Simple and multiple OLS. Fit the line, then add product category and read the full summary.
import statsmodels.formula.api as smf
simple = smf.ols("order_value ~ prior_30d_spend", df).fit()
print(f"simple R^2 = {simple.rsquared:.3f}") # ~ 0.36, matches r^2
model = smf.ols("order_value ~ prior_30d_spend + C(product_category)", df).fit()
print(model.summary()) # coefficients, R^2, p-values, CIs
print(f"multiple R^2 = {model.rsquared:.3f}")
# residual check: a shapeless cloud around 0 is healthy
import matplotlib.pyplot as plt
plt.scatter(model.fittedvalues, model.resid, s=6, color="#7FD1DE")
plt.axhline(0, color="#F97362"); plt.xlabel("fitted"); plt.ylabel("residual")
plt.title("Residuals vs fitted"); plt.show()
Fallacy 1 - p-hacking, and the fix. Correlate 20 pure-noise features with order value. About one clears p < 0.05 by luck alone. Then correct for the 20 looks.
from statsmodels.stats.multitest import multipletests
rng = np.random.default_rng(7)
pvals = []
for _ in range(20): # 20 features that are pure noise
noise = rng.normal(size=len(df))
_, p = stats.pearsonr(noise, df.order_value)
pvals.append(p)
pvals = np.array(pvals)
print("raw hits (p<0.05):", int((pvals < 0.05).sum())) # ~ 1 by chance
bonf = multipletests(pvals, alpha=0.05, method="bonferroni")[0]
fdr = multipletests(pvals, alpha=0.05, method="fdr_bh")[0]
print("after Bonferroni:", int(bonf.sum())) # ~ 0
print("after FDR (BH) :", int(fdr.sum())) # ~ 0
# Test 20 things at alpha 0.05 and ~1 fake "win" is EXPECTED. Correct for the count.
This is the single most common way dashboards lie. A growth analyst slices conversion by 20 audience traits, one comes back "significant", and a whole campaign is built on noise. The defense is not intuition - it is multipletests. Bonferroni divides your alpha by the number of tests (strict); FDR controls the expected share of false discoveries (more forgiving at scale). Either one erases the phantom win.
Fallacy 2 - Simpson's paradox on Lumen. A channel that looks better overall is worse inside every product category once you group. Watch the verdict reverse.
# Two channels, conversion counts by product category (authored to reverse).
# 'social' looks better OVERALL, yet loses WITHIN every category.
data = pd.DataFrame([
# channel category conversions sessions
["social", "cleanser", 18, 1000],
["social", "serum", 90, 3000], # social sits mostly in easy categories
["email", "cleanser", 75, 3000],
["email", "serum", 36, 1000], # email sits mostly in hard categories
], columns=["channel","category","conv","sessions"])
overall = data.groupby("channel").apply(
lambda g: g.conv.sum() / g.sessions.sum())
print("OVERALL rate:\n", (overall*100).round(2)) # social ~2.70% > email ~2.78%? check below
bycat = data.assign(rate=lambda d: (d.conv/d.sessions*100).round(2))
print("\nWITHIN each category:\n",
bycat.pivot(index="category", columns="channel", values="rate"))
# Within cleanser AND serum, email's rate is HIGHER - the aggregate flipped it
# because social's volume piles into the easy-converting category. Group before you rank.
Simpson's paradox is not a math curiosity - it is why "channel X has the best overall conversion" can be exactly backwards. The aggregate is a weighted average, and the weights (how much traffic each channel sends to easy vs hard categories) can dominate the real per-segment performance. The habit that saves you: before you rank anything, ask what confounder the groups differ on, and group by it.
Your capstone this week ◐ 60 min total
- Fit and defend a model. Build a multiple OLS predicting
order_valuefrom prior spend and category, report R-squared, and write two sentences a stakeholder would trust - including one caveat about what the model does NOT prove (causation). - Reproduce Anscombe. Load
seaborn.load_dataset("anscombe"), compute Pearson r for all four sets (nearly identical), then plot them. Write the one-line lesson. - Scale the p-hack. Push the noise loop to 100 features. How many raw hits now, and does FDR keep more than Bonferroni? Explain the trade-off in one sentence.
- Author your own Simpson's reversal. Change the volumes in
simpson.pyuntil the overall winner and the per-category winner disagree, and name the confounder that drives it.
Three questions before you go 🎯 ◐ 90 seconds
1 · A dataset has Pearson r ~ 0 but a clear U-shape in the scatterplot. What is true?
Pearson r captures linear association only. A strong curve can average out to r near 0. Always plot; consider Spearman or a nonlinear model when the shape is not a line.
2 · You test 20 unrelated features against an outcome at alpha 0.05. About how many "significant" hits do you expect by chance?
At alpha 0.05, each test has a 5% false-positive chance, so 20 tests yield about one by luck. Multiple-comparison corrections (Bonferroni, FDR) rein that in.
3 · A channel has the best overall conversion but the worst rate inside every product category. What is going on?
Simpson's paradox: the overall rate is a weighted average, and uneven weights (a channel concentrated in easy-converting categories) can flip the aggregate against the per-segment truth. Group by the confounder before ranking.
What this session covers
This capstone covers ~80% of the introductory correlation and regression content in the sources below, on the Lumen dataset - plus the fallacy defenses that most intro courses skip.
Builder Session 10 cheat sheet · pin this
stats.pearsonr (linear), stats.spearmanr (monotonic, rank-based, robust).smf.ols("y ~ x + C(cat)", df).fit(); read .summary(), R-squared, residuals vs fitted.multipletests(..., method="bonferroni"|"fdr_bh").