Why this is the highest-stakes session
Nobody lies to you with raw data - they lie to you with summaries. An average here, a truncated axis there, a "correlation" quietly promoted to a "cause", and a slide can steer a serious budget while being technically true and completely misleading. The defense is not a statistics degree; it is a short list of questions asked in the right order. Today: describe before you decide, interview the dataset before modeling it, and never confuse "moves together" with "causes".
Describe before you decide 5 min live
Every dataset gets compressed before it reaches you: a center (mean, median, mode), a spread (range, percentiles, standard deviation), and a shape (the distribution). Each compression throws information away - the skill is knowing WHICH information just got thrown away, because that is where the surprises hide.
LiveWhy "average" is the most dangerous word in business3 min▶
- Mean adds everything and divides - democratic, but one whale drags it anywhere. Median is the middle customer - whales cannot move it. Mode is the most common value - the "typical basket" nobody computes but everyone imagines.
- Rule of thumb: money data (deal sizes, salaries, order values, customer spend) is almost always skewed by a few giants. For skewed data, the median tells the truth about "typical"; the mean tells the truth about "total divided by heads" - two different questions.
- Spread is strategy. Two products with the same average delivery time - one always 5 days, one ranging 1 to 20 - need completely different promises to customers. The average is identical; the business is not.
- Outliers are not dirt. Sometimes they are typos to remove; sometimes they are your next enterprise segment announcing itself. Ask which before anyone "cleans" them away.
The US Census Bureau publishes household income both ways, and the gap is the lesson: mean household income runs roughly 40% higher than median, because a small number of very high earners drag the mean up. Report the mean and the country looks richer than the typical household actually is - which is exactly why economists, and sharp executives, reach for the median first on any money metric.
Self-studyPercentiles: the executive's favorite spread measure2 min read▶
Standard deviation is fine, but percentiles speak human. "Median delivery is 3 days, 90th percentile is 11" means: half of customers get it in 3 days, but one in ten waits 11 or more - and that unlucky 10% writes your reviews. Tech companies run on this grammar (p50, p95, p99 response times) because averages hide exactly the customers who are angriest. When you set service targets, set them at a high percentile, not at the mean - the mean has never waited for a parcel.
EDA: the walk around the building 4 min live
Exploratory data analysis (EDA) is the discipline of interviewing a dataset before trusting it with a decision - profile it, question it, plot it, form hunches - BEFORE any model or metric gets built. John Tukey, the Bell Labs statistician who coined the practice in the 1970s, compared it to detective work: evidence first, verdict later.
LiveThe four moves of EDA (and the chart-choice mini-guide)4 min▶
- Profile: how many rows? what date range? how much is missing? A churn analysis where 30% of customers have no signup date is a different project.
- Question: what would surprise me here? Write it down before looking - it keeps you honest when the data confirms what you hoped.
- Visualize: plot the distributions, not just the summaries. Shapes reveal what averages conceal - two humps, a whale tail, a cliff at a suspicious round number.
- Hypothesize: end EDA with sentences that start "it looks like..." - candidates for testing, not conclusions for slides.
And when someone (including you) picks a chart, match it to the question:
| Question | Right chart | Watch out for |
|---|---|---|
| Compare categories | bar chart | truncated axes exaggerating small gaps |
| Trend over time | line chart | cherry-picked start dates |
| Shape of the data | histogram | bin sizes chosen to hide the whale |
| Relationship of two things | scatter plot | a trend line drawn through noise |
| Share of a whole | bar, NOT pie if over 4 slices | humans cannot compare pie angles |
Self-studyAnscombe's quartet: same stats, four different truths2 min read▶
In 1973 statistician Francis Anscombe built four small datasets with essentially identical summary statistics - same means, same variances, same correlation, same trend line. Then he plotted them: one is a clean linear relationship, one is a curve, one is a straight line bent by a single outlier, and one is a vertical stack where a lone point invents the entire "relationship". Four identical spreadsheets of summary stats, four wildly different realities. It is the most elegant argument ever made for one habit: always plot the data. Any analysis you are shown that contains only summary numbers and zero pictures has skipped the step Anscombe built his quartet to defend.
Correlation, causation and the paradox 6 min live
Ice cream sales and drownings rise together every year. Ice cream does not drown anyone - summer causes both. "Moves together" is cheap to find and easy to sell; "causes" is expensive to prove and is the only thing worth acting on. Between the two sits the confounder: the hidden third factor pulling both strings.
LiveConfounders, the Berkeley case, and the A/B instinct4 min▶
- The confounder question: for any "X drives Y" claim, ask "what third thing could drive both?" Summer drives ice cream AND drownings. Company size drives tool adoption AND revenue. It deflates half of all correlation claims in one sentence.
- Simpson's paradox is the confounder's masterpiece: a trend that holds in every group can reverse when the groups are blended, because the group mix is doing the driving.
UC Berkeley, 1973: graduate admissions looked biased - 44% of men admitted vs 35% of women. The graduate division feared being sued and commissioned an analysis; no lawsuit was ever filed, though the lawsuit version of the story is repeated in a great many textbooks. Broken down BY DEPARTMENT the picture changed: the published analysis found four departments significantly biased against women and six significantly biased against men, with the pooled figure showing a small bias in favour of women. The confounder: which departments people applied to. Women disproportionately applied to fiercely competitive departments with low admission rates for everyone; men flocked to high-admission ones. The blended number was arithmetically true and directionally wrong - the canonical warning taped above every dashboard of blended averages. The 1975 paper itself would not load when these figures were last checked, so they come from three independent sources that reproduce its tables and agree to the digit.
- So how do you ever PROVE causation? Run an experiment. Split customers randomly, change one thing for one group, compare. Randomization silently cancels every confounder - known and unknown - which is why A/B testing is the only clean causal instrument business has.
Bing's $100M title tweak: Ronny Kohavi's experimentation team at Microsoft tested a tiny change to how ad titles displayed - an idea that had sat in the backlog for months because nobody expected much. The A/B test revealed it was worth roughly $100M a year in revenue. The honest twin of the story: Kohavi reports that most tested ideas fail to move their target metric - at Bing, about two-thirds. Experts cannot tell the $100M ideas from the duds by intuition. That is not an argument against experts; it is the argument FOR testing.
Self-studySampling: who is missing from this data?2 min read▶
Every dataset is a sample of some reality, and the fastest way to lie without lying is to sample the convenient. Survey respondents are people who answer surveys. Churned customers do not fill in exit polls. The classic: in WWII, analysts mapped bullet holes on returning bombers to decide where to add armor - until statistician Abraham Wald pointed out the holes marked where planes could be hit AND STILL COME HOME. Armor the clean spots: the fatal hits are on the planes you never see. Before trusting any dataset, ask the Wald question: "who is not in this data, and why?"
Why the average lies ★ 7 min · everyone plays
Ten ordinary orders, humming along. Then sales closes one enterprise whale. Watch what happens to each summary number - and decide which one belongs in the board pack.
Before clicking: predict. The whale is worth $1,800 against orders around $100. How far will the mean jump? Will the median move at all? Say your guesses out loud.
Click "Add the whale client". Check your predictions - most people underestimate the mean's jump and overestimate the median's.
Debate: which number goes in the board pack as "typical order value"? The honest answer is usually the median, PLUS the whale reported separately by name - never the blended mean pretending the whale is ten normal customers.
Translate to your own business: name your whale metric - the one number you report where a few giants drag the average. Deal size? Order value? Time-to-close?
Stat-proof one slide ★ 8 min · everyone audits
Pick a real claim from a slide you saw recently: "satisfaction is up", "the campaign worked", "churn improved". Write it verbatim.
Run the four challenge questions: Versus what baseline? (up from when, compared to what?) Mean or median? (and would the other tell a different story?) How many data points? (12 survey responses is an anecdote wearing a chart) Could a confounder explain it? (seasonality, a price change, one big customer?)
Rewrite the claim honestly. "Satisfaction up" often becomes "median satisfaction flat; mean lifted by 3 enthusiastic enterprise accounts; n=41". Less triumphant, infinitely more useful.
Share the best before/after rewrite with the room. The gap between the two versions is exactly what this session is for.
The correlation courtroom ★ 6 min · everyone votes
Four claims, straight from the wild. For each: the room votes causal or not-proven, then names the most plausible confounder. No claim leaves without a suspect.
Claim 1: "Users of feature X churn less - X reduces churn." Vote, then find the confounder. (Engaged power users both adopt features AND stay. Does X retain them, or do the retained use X?)
Claim 2: "Stores with more staff sell more - hire more staff." Vote, confounder. (Busy high-traffic locations get more staff AND more sales. Traffic drives both.)
Claim 3: "Customers who contact support churn more - reduce support contact." Vote, confounder. (Product problems drive both the ticket and the churn. Muzzling support hides the smoke, not the fire.)
Claim 4: "Trial users who invite a colleague convert 3x more - prompt invites harder." Vote, confounder - then discuss: this is exactly the kind of claim worth an A/B test, because if it IS causal, it is a growth lever.
This week ◐ 25 min total
- Ask "what is the median?" once in a real meeting where an average is presented. Note the reaction - and whether the answer changes the decision.
- Finish your stat-proofed slide from Apply-along 2 and send the rewrite to its original author, kindly. You will make one ally and one enemy; both are progress.
- Read the self-study cards: percentiles, Anscombe's quartet, and the Wald sampling story.
- Optional deep end: Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments, chapter 1 - the best executive briefing on experimentation ever printed.
Three questions before you go 🎯 ◐ 90 seconds
1 · You are reporting "typical customer spend" and the data contains a few giant enterprise accounts. Which center do you reach for?
Money data is almost always whale-skewed. The median answers "what is typical"; the mean answers "total divided by heads" - a different question. Blend the whale into the mean and you describe a customer who does not exist.
2 · Simpson's paradox means...
Berkeley 1973: the blended total looked biased against women, yet the department-level analysis found the bias running both ways and slightly in favour of women overall - the mix of departments applied to, not the departments themselves, produced the aggregate. Always ask "does this hold inside each segment?"
3 · What can a well-run A/B experiment tell you that no amount of historical correlation analysis can?
History can only show what moved together; a lurking third factor can always explain it. Random assignment breaks every such backdoor at once - which is why Bing tested its way to a $100M title change that intuition had shelved for months.
Concepts covered & their origins
The working 80% of each idea, credited to its canon - depth on demand for the curious.