Summaries are lossy - know what you dropped
A dataset of 50,000 Lumen orders becomes, in one line of pandas, a handful of numbers. That compression is the point - and the danger. df.describe() hands you a mean without a warning that the data is skewed, a standard deviation without telling you the distribution has a fat tail. This session makes you fluent in computing descriptive statistics and suspicious of them in exactly the right places.
mean, median, std, percentile, describe, skew - plus histograms and boxplots in matplotlib. Crucially, you will see the mean-vs-median gap appear in real numbers and know how to report it honestly.
Center, spread, shape - and the functions that compute them 6 min live
Three questions, and the numpy/pandas call for each. The whole of descriptive statistics fits on one card - the skill is knowing which number to trust on which data.
LiveThe functions, and what they compute3 min▶
- Center:
np.mean/df.mean(),np.median/df.median(),df.mode(). - Spread:
np.std/df.std()(note the ddof gotcha below),np.percentilefor quartiles, IQR = Q3 - Q1. - Shape:
df.skew()(> 0 is right-skewed),df.kurt(), and always a histogram to see it. - All at once:
df.describe()gives count, mean, std, min, 25/50/75%, max.
numpy.std defaults to ddof=0 (population SD, divide by N). pandas.std defaults to ddof=1 (sample SD, divide by N-1). On a sample you almost always want N-1, so they disagree unless you set ddof explicitly. We unpack why N-1 in Builder Session 6.
Build Lumen once, describe it honestly 22 min · everyone builds
We simulate the Lumen orders with a fixed seed so your numbers match mine - and match every later builder session. Right-skewed order values (lognormal), a ~3.2% conversion flag, and a prior-spend column we will need for correlation in b10.
Build the dataset (seed 42). Run this once; save it - b2 through b10 reload it.
import numpy as np, pandas as pd
rng = np.random.default_rng(42)
N = 50_000
# order_value: right-skewed money, lognormal, mean ~ $74, median ~ $63
order_value = np.round(np.exp(rng.normal(4.0, 0.55, N)), 2)
order_value = np.clip(order_value, 5, 400)
df = pd.DataFrame({
"order_value": order_value,
"product_category": rng.choice(
["cleanser", "serum", "moisturizer", "set"], N, p=[.34, .28, .26, .12]),
# prior-30-day spend: correlated with this order (rho ~ 0.6) - used in b10
"prior_30d_spend": np.round(
0.6 * order_value + 0.8 * rng.normal(74, 38, N), 2).clip(0, None),
})
df.to_parquet("lumen_orders.parquet") # reload in later sessions
print(df.shape)
print(df.head())
Describe it - then look past describe(). The mean/median gap is the whole lesson.
ov = df["order_value"]
print(df.describe()) # count, mean, std, min, 25/50/75%, max
print("mean :", round(ov.mean(), 2)) # ~ 74
print("median :", round(ov.median(), 2)) # ~ 63 -> mean > median
print("std :", round(ov.std(), 2)) # pandas: ddof=1 (sample)
print("IQR :", round(ov.quantile(.75) - ov.quantile(.25), 2))
print("p90 :", round(ov.quantile(.90), 2))
print("skew :", round(ov.skew(), 2)) # > 0 confirms the right tail
Prove the ddof gotcha to yourself. Same data, two "standard deviations".
import numpy as np
print("numpy ddof=0 :", round(np.std(ov), 4)) # population
print("numpy ddof=1 :", round(np.std(ov, ddof=1), 4)) # sample - matches pandas
print("pandas default:", round(ov.std(), 4)) # ddof=1
See the shape. A histogram and a boxplot make the skew undeniable.
import matplotlib.pyplot as plt
fig, ax = plt.subplots(1, 2, figsize=(11, 4))
ax[0].hist(ov, bins=60, color="#7FD1DE", edgecolor="white")
ax[0].axvline(ov.mean(), color="#F97362", ls="--", label=f"mean {ov.mean():.0f}")
ax[0].axvline(ov.median(), color="#0E7C90", label=f"median {ov.median():.0f}")
ax[0].legend(); ax[0].set_title("Order value - right-skewed")
ax[1].boxplot(ov, vert=False)
ax[1].set_title("Same data as a boxplot")
plt.tight_layout(); plt.show()
A Lumen analyst shipped a dashboard headline: "avg order $74." The growth team built a persona around it. The histogram above would have stopped them - the mode is nearer $50, the median $63, and the $74 is a tail artifact. One ov.skew() of about +0.9 is the number that says "report the median too."
This week ◐ 40 min total
- Group and compare. Use
df.groupby("product_category")["order_value"].agg(["mean","median","std"]). Which category is most skewed (mean furthest above median)? - Robust vs sensitive. Append one $10,000 order to the data. Re-check mean, median, std, and IQR. Which barely moved? That is robustness.
- Percentiles by hand. Confirm
np.percentile(ov, 90)matchesov.quantile(.90), and write one sentence on what "p90" means to a non-technical stakeholder. - Optional: add a boolean
convertedcolumn at ~3.2% true withrng.random(N) < 0.032- we build on it in b2.
Three questions before you go 🎯 ◐ 90 seconds
1 · np.std(x) and pandas Series.std() give slightly different numbers on the same data. Why?
numpy uses the population formula (ddof=0) by default; pandas uses the sample formula (ddof=1). On a sample you want N-1, so set ddof explicitly to make them agree.
2 · df.describe() reports mean, std, and quartiles but NOT which of the following?
describe() covers center and spread (count, mean, std, min, 25/50/75%, max) but not shape. Call df.skew() and plot a histogram to judge skew.
3 · You append one huge $10,000 order to Lumen's data. Which pair barely moves?
The median and IQR are built from order positions, so a single extreme value barely shifts them. The mean and standard deviation are built from the actual values, so they lurch toward the outlier.
What this session covers
This builder session covers ~80% of the computational descriptive-statistics content in the sources below, on the Lumen dataset. Deeper theory (why N-1, the shape of estimators) arrives in b6.
Builder Session 1 cheat sheet · pin this
df.mean(), df.median(), df.mode(). On skewed money, report the median.df.std() (ddof=1), np.percentile, IQR = q75 - q25. SD pairs with mean, IQR with median.df.skew() > 0 = right tail. describe() will NOT tell you this - always plot.np.random.default_rng(42). Fixed seed = your numbers match the course all the way to b10.