learn-sql-with-phoebe / Leader session 3 of 6
Learn SQL with Phoebe · Leader track · Session 3 of 6

Judging a data claim

Most misleading numbers are not lies - they are honest queries that quietly answered a narrower question than the slide implies. This session trains the reflex that separates a sharp leader from a nodding one: when someone says "the data shows", you can ask exactly what the query counted, per what, and what it left out. Read the query, spot the trap, then decide.

🟡 Leader track C-level · managers · curious non-coders No install · nothing to write 45 min
0-3 · Recap 3-21 · Granularity & summaries 21-39 · Where numbers mislead 39-45 · Q&A
Part 0

Where we are

You can now read a query and say what it asks for. This session puts that reading to work as judgment. A single number - an average, a total, a rate - is a lossy summary of a richer table, and the way it was computed decides whether it deserves your trust. We will look at real Daybreak numbers that are each true and each quietly misleading, and practice the three questions that expose the gap.

Live - discussed in session Self-study - read after ▶ See it run - a real query, live For decision-makers
★ What you walk out with today A gut feel for hidden granularity ("per what?"), the knowledge that count, sum and average tell different stories from the same data, and the three questions to ask of any metric before it drives a decision.
Part 1 · the hidden 'per what'

GROUP BY = per what: the hidden granularity 6 min live

Every metric is secretly "per something". "We have customers" becomes a very different statement once you ask "per country". The GROUP BY clause is where that granularity is decided, and it is the first thing to locate when you judge a claim. Change what you group by and the same table tells a different story.

LiveCustomers, per country3 min

Read it first: "count of customers, per country, biggest first." The GROUP BY country is the granularity; the ORDER BY ... DESC just sorts the summary. Run it, then ask: would a headline of "we serve five countries" have told you the same thing? It would have hidden that the count is lopsided.

GROUP BY country: one row per country, its own COUNT(*) - 15 customers total USA 8 customers Canada 3 Germany 2 UK 1 Singapore 1 "We serve five countries" is true - and 8 of 15 customers sit in one of them. The grain exposes what the total hides.
🔍 Click to zoom - five countries is true, lopsided is the story
SELECT country, COUNT(*) AS customers
FROM customers
GROUP BY country
ORDER BY customers DESC;
The first question for any metric "Per what?" A total with no granularity hides how it is distributed. Finding the GROUP BY - or noticing there isn't one - is step one of judging a claim.
Self-studyWhy "per what" is the root of most disputes2 min read

When two teams argue about a number, they are usually grouping at different grains without realizing it. Sales counts revenue per deal; finance counts it per invoice; the product team counts it per active account. All three are "revenue", none agree, and none is lying. The fix is not a better database - it is naming the grain out loud.

  • Per order vs per customer: a customer with five orders is one row in one view and five in the other.
  • Per month vs per cohort: the same signups look like growth or churn depending on which you group by.
  • The leader's habit: before debating a number, agree on its grain. Half of data disputes dissolve right there.
Part 2 · one table, many stories

Count, sum and average tell different stories 6 min live

From one column you can compute several summaries, and each answers a different question. Count tells you how many, sum tells you the total, average tells you the typical - and average is the one that most often hides trouble. A single number is always a lossy compression of the rows behind it.

LiveThree summaries of the same price column3 min

Read it: "how many products, the average price rounded to cents, and the highest price." Run it. The catalog has coffee bags around $15-22 and one Cold Brew Kit at $34. Watch how the average lands in the middle and says nothing about that top-end outlier - a leader who only saw "average price" would never guess a $34 item existed.

8 prices, one average: AVG(price) = 19.88, MAX = 34.00 Cold Brew Kit 34.00 Ceramic Dripper 28.00 Single-Origin Ethiopia 22.00 Midnight Espresso 18.00 House Decaf Beans 17.00 Sunrise Blend 16.00 Decaf Calm 15.00 Oat Milk Pods 9.00 AVG = 19.88 The mean lands between the coffee bags. Anyone shown only "avg price 19.88" never learns a 34.00 item exists.
🔍 Click to zoom - the average sits where no product is, and hides the outlier
SELECT COUNT(*) AS products,
       ROUND(AVG(price), 2) AS avg_price,
       MAX(price) AS priciest
FROM products;
Real world

The average that hid the outlier. A pricing review anchored on "our average product is about $20" and set a discount policy around it. The equipment line - a handful of items at $28 and $34 - was where the margin actually lived, and the average had erased it from the conversation. The single number was correct and misleading at once. Asking to see the max and the spread, not just the average, would have caught it.

Self-studyWhen each summary is the honest one2 min read

None of these is better; each fits a different question. Match the summary to the decision.

SummaryAnswersHides
COUNThow many rows?the size or value of each
SUMwhat is the total?how it splits across rows
AVGwhat is typical?outliers and the shape of the spread

When you are handed an average, the follow-up is almost always "and what does the spread look like?" A median, a max, or a simple range often changes the story.

Part 3 · the common traps

Where numbers mislead 6 min live

Three traps account for most misleading metrics: the refund trap (a total that silently includes or excludes reversed transactions), the average that hides a lopsided distribution, and the mismatched denominator. The same dataset can support two opposite conclusions depending on which trap you fall into.

One dataset the same 33 orders Count all rows "33 orders - great quarter" Count completed only "fewer once refunds drop out" Same data, two conclusions. The metric you pick is the argument you make.
🔍 Click to zoom - one dataset, two honest but different conclusions
LiveOrders per status - the ambiguity, exposed3 min

Read it: "count of orders, per status." Run it and you will see the completed, refunded and cancelled buckets side by side. This is the whole point: "we had 33 orders" is ambiguous until you say which of these statuses were meant to count. The refund trap lives right here.

SELECT status, COUNT(*) AS n
FROM orders
GROUP BY status;
The refund trap A headline "orders" or "revenue" number is only meaningful once you know whether refunded and cancelled rows were kept or dropped. Always ask which statuses the total includes.
Self-studyThe denominator trap2 min read

Rates and percentages mislead through their denominator - the number on the bottom. "Conversion is up" means nothing until you know conversion out of what. A rate is only trustworthy when its top and bottom describe the same population over the same window.

  • Mismatched populations: numerator counts all signups, denominator counts only paying accounts - the rate is meaningless.
  • Shrinking denominator: a "churn rate improved" can simply mean the base you divided by got smaller.
  • The leader's move: for any rate, ask "top over bottom - are they the same group, same period?" Session a4 goes deep on this.
Boardroom exercise

Interrogate one claim as a group ★ 12 min · discuss together

Take the claim "our average order is healthy." Before running anything, the group writes down the three questions a leader should ask about how that number was computed. Then use the live queries to check whether the claim survives them.

Question 1 - what did it count? Did "order" mean every row, or completed orders only? Refunds in or out?

Question 2 - per what? Average per order line, per order, or per customer? Each gives a different "average".

Question 3 - what did it exclude, and what is the spread? Does one big outlier carry the average? Show the max, not just the mean.

LiveCheck A · what statuses are we averaging over3 min

Before you can trust an "average order", you have to know which orders count. Run this to see the status breakdown, then decide which rows the claim should have used.

SELECT status, COUNT(*) AS n
FROM orders
GROUP BY status;
LiveCheck B · average versus the outlier3 min

Run this to see the average price sitting next to the max. If the two are far apart, an outlier is shaping the "healthy average" - and the claim needs the spread shown, not just the mean.

SELECT ROUND(AVG(price), 2) AS avg_price,
       MAX(price) AS priciest,
       MIN(price) AS cheapest
FROM products;
Between sessions

Take this back to your desk ◐ 15 min

Source material

What this maps to

This session takes the aggregate-functions material from the standard curriculum and reframes it as judgment - how a summary can mislead - rather than syntax to memorize. It covers:

Mode SQL Tutorial - aggregate functions (conceptual level)Parts 1-2 · COUNT / SUM / AVG / MAX and GROUP BY granularity
Averages, outliers and denominators as failure modesParts 2-3 · why a true number can still mislead
Decision-intelligence framing (Cassie Kozyrkov style)Part 3 · the three questions to ask of any metric
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · "Our average price is $X" can mislead because...

An average is a lossy summary. A single $34 item can sit invisibly inside a "typical $20" number. Ask for the spread - a max, a min, a median - alongside any average.

2 · "We had 33 orders" is ambiguous because...

An order count is only meaningful once you know which statuses it includes. Completed only? Refunds in? The refund trap lives exactly here - ask which rows counted.

3 · The leader's core question about any metric is...

Three parts: what was counted, at what grain, and what was left out. Those questions turn a number you are shown into a number you can actually judge.

Leader session 3 cheat sheet · pin this

Every metric is "per something"find the GROUP BY, or notice there isn't one.
COUNT vs SUM vs AVGhow many, the total, the typical - three different truths.
Average hides the spreadalways ask for a max, a min, or a median beside it.
The refund trapa total is meaningless until you know which statuses it counts.
The denominator trapa rate is "X out of what?" - top and bottom must match.
Same data, two storiesthe metric you pick is the argument you make.
The three questionswhat did it count, per what, and what did it exclude?
Next sessiona4 · the dashboard behind the dashboard - every tile is a query.