Asking the right questions of your DA

Here is the same finding presented twice. The first version happens in real meetings every week; the second is what you should insist on.

• Bad readout: "Good news - retention improved to 34% this quarter." Sounds like progress. But 34% of WHOM? All signups or activated users? Which months? Compared to what baseline, measured at what cohort age? And is 5 points worth one engineer-week or twenty? None of it is answerable from the sentence, so the meeting fills the gap with mood.
• Good readout: "The March signup cohort from paid social - 12,400 users - hit 34% month-1 retention versus 29% for February's cohort at the same age, after the new onboarding shipped. That 5-point gain, times cohort size, times $18 average monthly revenue, is worth about $11k per month if it holds. We will know it holds when April's cohort matures." Every clause answers one of the three questions, and the last sentence pre-registers the evidence.

Notice what the good version makes possible: you can now DECIDE something - roll the onboarding change to other channels, or wait for April. The bad version only lets you feel something. The three questions are how you convert charts into decisions:

• WHICH cohort: month, channel, product, and the denominator rule (all signups or activated users). This pins the number to a population that cannot quietly change.
• WHAT changed: versus the comparison cohort at the SAME age, and what event explains the gap. No comparison, no story.
• HOW MUCH money: delta retention x cohort size x ARPU. This is what ranks the finding against everything else competing for the roadmap - exactly how Duolingo decided CURR was the metric worth four years of work.

Analysts sometimes hear the money question as an attack. It is the opposite - it is what gets their work funded. A retention finding without a dollar figure competes for attention against every loud opinion in the room; a finding WITH one competes on the roadmap, where analysis usually wins.

• The formula is deliberately crude. Delta x cohort size x ARPU ignores compounding, margins, and second-order effects - and that is fine. It is a ranking device, not an accounting entry. Two significant figures is plenty.
• Ask for the range, not the point. "Worth $8k-14k a month depending on whether the lift holds at month 3" is more honest and more useful than false precision.
• Make it standard, not personal. The healthiest version is a template your team always uses (the one-pager above), so no analyst feels singled out when you ask where box 3 went.

What you just drove is Simpson's paradox wearing a business suit: a trend that appears in a combined number can reverse when you split the number into the groups that make it up. Nothing in the dashboard was wrong - every figure was computed correctly. The lie was structural, not arithmetic.

• The mechanism in one line: blended retention is a weighted average, and the weights are the acquisition mix. Change the weights and the average moves - no user behavior required.
• It cuts both ways. A genuinely improving product can show FALLING blended retention if acquisition shifts toward a weaker channel - so the trap punishes good teams too, not just credulous ones. Decomposition is protection in both directions.
• The question to institutionalize: no retention trend gets reported without its decomposition by the segments that could plausibly have shifted - channel, geography, platform, plan. Your analyst can produce this in minutes; they mostly are not asked.
• Connect it to the three questions: "WHICH cohort" already contains the antidote. A readout that names "March, paid social" cannot mix-shift on you, because it is a single segment. Blends are where the trap lives.

• Denominator games. "Retention" computed on all signups and "retention" computed on activated users (people who completed a first core action) are DIFFERENT questions - the second reads far higher and measures product stickiness, while the first measures funnel plus product together. Both are legitimate; swapping between them silently is not. The ask: "which denominator, and has it changed since last quarter?"
• Calendar periods vs cohort ages. "March actives vs January actives" is not retention - it mixes cohorts of every age with seasonality and acquisition volume. Retention compares cohorts at the SAME age: January's month-2 versus February's month-2. If the chart's x-axis is calendar months and it is being called retention, send it back.
• D1 lifts that do not propagate. A new welcome screen lifts day-1 retention - lovely. But if the D30 curve is unchanged, you moved the greeting, not the habit. The ask: "does the lift survive to day 30, or do the curves reconverge?"
• Dashboards whose history restates - the frozen-history law. Pull up last quarter's number. If it is different from what you reported last quarter, the metric is rolling or trailing: its definition re-anchors every day, so it can never be trended, compared, or targeted against. This one is not a caveat - the metric should be replaced with frozen forward-looking cohorts.
• Immature cells read as decline. The newest cohort always looks worst, because its later windows have not happened yet. "Not yet" is not "churned". The ask: "are the youngest cells marked partial, and are they excluded from averages?"

You can catch most of the list without asking a single question, just from how the chart is drawn. A pocket field guide:

• The y-axis label says "retention" with no cohort or age in sight: suspect calendar-vs-cohort or a trailing window. A trustworthy retention chart names the cohort dimension somewhere.
• The trend is suspiciously smooth and only goes up: suspect a blend hiding a mix shift, or a rolling metric quietly restating its own past upward. Real segment-level retention is lumpy.
• The newest period is the lowest bar on the chart: almost always an immature cell being read as decline. Check whether it is marked partial.
• Footnotes changed but the metric name did not: denominator swap. This is why the readout template makes the denominator part of "WHICH cohort".
• The healthiest habit of all: keep one old readout in a drawer. Any time a dashboard shows the same period, compare. If history moved, you have caught a restating metric red-handed - no analytics skills required.