learn-product-analytics-with-phoebe / Leader session 3 of 6
Learn Product Analytics with Phoebe · Leader track · Session 3 of 6

Engagement you can defend

"Engagement is up" is the sentence most likely to be said in a product review and least likely to survive a follow-up question. Tonight we take the three follow-ups in order: which ratio you quoted and what it hides, what shape the users are actually in, and which definition of "active" produced the number. Every count comes from 79,555 real GitHub accounts watched for 28 days, and the sampling that produced them is on the page, because a number you cannot explain is a number you cannot defend.

🟡 Core Product managers and product leaders No code 45 min
0-3 · Welcome 3-18 · The ratio, the curve, the definition 18-40 · Build-along: your engagement definition card 40-45 · Q&A
Part 0

Where we start

Session 1 wrote the diary. Session 2 found the moment that predicts staying. Tonight is about the number that gets quoted most and questioned least: engagement. Most teams report it as one ratio, daily actives over monthly actives, and the ratio is not wrong, it is just quiet about the thing that matters, which is how many different kinds of user are hiding inside it. A product with a small core that shows up every day and a product where everyone drops in twice a month can print the same DAU/MAU. The way to see the difference is a histogram, not a ratio, and the way to trust either is to know exactly which action made someone count as active.

🎫 Clear all three quiz questions to stamp your passport Live = presented in the 45 minutes Self-study = the full depth, after class ★ Build-along = you make the artifact

★ What you walk out with tonight. A one-page engagement definition card for your product: the one action that means someone got value, the written definition of "active" built on it, the period (daily or weekly) that matches how often the product is naturally used, and a list of everything that would inflate the number without anyone getting value. Plus the habit of asking, before any engagement chart is discussed, "what shape is the curve, and what counted as active?"

Part 1 · covers DAU/MAU and why one ratio blurs

The ratio that hides the shape

5 min live

DAU/MAU is the number of daily active users divided by the number of monthly active users, usually averaged over the month. It is quoted as "stickiness": if 100 people used the product this month and 20 used it on a typical day, the ratio is 20 percent. It is a fine number. The trouble is what Li Jin and Andrew Chen put in one clause in 2018: "it's a single number and so blurs this variance". Two products with the same ratio can be made of completely different people.

TWO PRODUCTS, ONE RATIO a sketch of the idea: bar heights are drawn, not measured Product A: everyone drops in 1 day days active 30 Product B: visitors plus a daily core 1 day days active the core same DAU/MAU The ratio averages both shapes into one number. Only the histogram shows that B has a core and A has none. That histogram is the power-user curve, and it is Part 2.
🔍 Click to zoom - the gold tag is the problem: the same ratio, and only the blue bar on the right tells you which product has a habit
LiveWhat DAU/MAU measures, and what it averages away3 min▶

The definition. Count the accounts active on each day, average that across the month, and divide by the accounts active at least once in the month. On our GitHub sample, with "active" meaning any event at all, average DAU is 5,110.6, MAU is 79,555, and the ratio is 6.4 percent. The weekly version, DAU/WAU, is 18.6 percent. Both are real, both come from the same accounts, and neither says what a typical account looks like.

Why a ratio blurs. Jin and Chen's point is arithmetic, not opinion. The ratio is a mean, and a mean of a mixed population hides the mix. Product A in the sketch could reach the same ratio as product B by having many people show up a few days each; product B gets there with a small group that shows up every day and a crowd that barely does. One of those products has a habit inside it and the other does not, and that is the first thing a product leader needs to know.

What to ask instead. "How many days in the month were people active, and how is that distributed?" That question has a chart, the histogram in Part 2, and the chart cannot be summarised by one number without losing the shape.

Self-studyThe period should match the product2 min▶

DAU/MAU assumes a product that could plausibly be used daily. Many products cannot be: an expense tool is opened at month end, a tax product once a year, a code-review tool on working days. For those, a daily ratio reads as failure no matter how healthy the product is, and the honest choice is a period that matches the natural frequency: weekly actives over monthly actives, or monthly actives over quarterly. The build-along asks you to pick that period on purpose, in writing, so that nobody later compares a weekly product to a daily one and calls it a decline.

One seam to note. Whether those actives are still here in six months is retention, and it has its own course: Customer retention with Phoebe. Tonight stays inside the month.

Real world

The stickiness slide, common pattern. A team reports DAU/MAU rising for three quarters and takes it as a habit forming. What actually happened, once someone drew the histogram: a batch of low-frequency accounts had stopped using the product, MAU fell, and the same daily core divided by a smaller month looked stickier. Nothing improved; the denominator shrank. No numbers here because it is a pattern rather than one company's case, and it is the most common way a ratio flatters a product that is losing its edges.

Part 2 · covers the power-user curve, L30 and the real L28

Draw the histogram, not the ratio

6 min live

Jin and Chen call it the power user curve: "a histogram of users' engagement by the total number of days they were active in a month". It is "also commonly called the activity histogram or the 'L30' (coined by the Facebook growth team)". The shape they hope for is a smile: a bar on the left for the occasional, a dip, then a bar on the right for the people who show up nearly every day. We drew one from 28 real days of GitHub, and it does not smile. Part of that is true of GitHub and part of it is how we sampled, and the page says which is which.

THE REAL L28: 79,555 GITHUB ACCOUNTS, MARCH 2024 active = any event in a sampled hour; share of accounts 66.7% 24.0% 7.5% 1.6% 0.2% 0.1% 1 day 2-3 days 4-7 8-14 15-21 22-28 days active out of 28 the smile a growth team hopes for (dashed) is not here 2 h a day sampled Median: 1 day. 62 accounts seen on all 28 days. The last two bars are drawn at a visible minimum height. Sampled: 5 percent of accounts, 03:00 and 15:00 UTC only. A daily user active at other hours looks occasional.
🔍 Click to zoom - the gold bar is two thirds of every account; whether that is GitHub or the two-hour window is the question the note answers honestly: both
LiveReading a power-user curve3 min▶

The x axis is days active in the period, 1 to 30 for a month, 1 to 28 here because we watched four full weeks. The y axis is the share of monthly actives at each count. Every bar together adds to MAU, so the curve contains DAU/MAU and more: the ratio is roughly the average bar position divided by the period length, and the curve is the whole distribution the ratio was averaged from.

The smile. Jin and Chen's ideal is a right-leaning smile: a meaningful bar at the far right, meaning a group that shows up almost every day. They write that Facebook "would have a very right-leaning smile, with 60%+ of its MAUs coming back daily". That is their claim about a company they did not work for at the time and it is not verifiable from outside, so treat it as an illustration of the shape, never as a benchmark to be measured against.

Our curve leans left. 66.7 percent of the sampled accounts were seen on exactly one day in 28; 24.0 percent on two or three; 7.5 on four to seven; and everything beyond 8 days is under 2 percent combined. There is no second bump. Two explanations are true at once, and the next card separates them.

LiveReal, or an artefact of how we looked2 min▶

The real part. Most public GitHub accounts are occasional. Someone stars a repository, opens one issue, pushes one fix to a fork, and is not seen again that month. Session 2 of the analyst track found the same thing inside a single hour: 52.1 percent of actors had exactly one event. A left-leaning curve is what a broad, mostly casual user base looks like, and there is nothing wrong with it as long as nobody presents it as a smile.

The sampling part. We watched two hours a day, 03:00 and 15:00 UTC, and only accounts whose id divides by 20. An engineer who pushes code every working day from 09:00 to 17:00 in San Francisco is invisible at both of our hours and lands in the 1-day bar or not at all. So the true GitHub curve has more weight to the right than this one, and by an amount we cannot compute from this sample. That sentence is the caveat that makes the chart defensible: it says what the number is, how it was made, and which direction it is biased.

The habit worth copying: every engagement chart carries its sampling and its definition in the footnote. If yours came from the full event stream, say so. If it came from a client library that some users block, say which direction that pushes it.

Self-studyConcentration is the other view of the same curve2 min▶

A left-leaning curve means the days are concentrated in few hands. In the sample, the top 10 percent of accounts hold 33.5 percent of all account-days, and the top 1 percent by event count hold 30.1 percent of all sampled events. Those two numbers are the same story as the histogram from a different angle: a small group does a disproportionate share, and the ratio would have hidden that too. Analyst session 5 builds this curve from the rows and shows where automation hides in its right tail, which is where Part 3 goes next.

Real world

62 accounts, 28 out of 28 days. In a sample of 79,555, exactly 62 accounts were seen in at least one of our two hours on every single day of March 2024. That is 0.08 percent. Before you call them your most loyal users, look at the names: the right tail is where scheduled jobs live, and the same 28-day streak that marks a devoted human also marks a cron job that runs at 03:00. Part 3 does the crude check.

Part 3 · covers vanity versus signal and the definition of active

Which "active" did you mean

4 min live

Every engagement number rests on a sentence nobody reads aloud: what counted as active. Change the sentence and the number moves, on the same accounts, from 79,555 to 8,993. A vanity metric is one the team can raise without any user getting more value; a signal is one that only moves when they do. The definition of active is where a metric becomes one or the other.

SAME ACCOUNTS, THREE DEFINITIONS, THREE MAUs THE RIGHT TAIL, CHECKED any event MAU 79,555 DAU/MAU 6.4% pushed code MAU 48,946 DAU/MAU 6.6% opened an issue or PR MAU 8,993 DAU/MAU 5.0% the two inner sets overlap: many who opened an issue also pushed accounts seen on 22+ of 28 days 105 accounts, the far right of the curve 17 have bot-like names, no [bot] suffix crude name check, nothing more: bot · -ci · ci- · sync · deploy · commenter · assistant The gold set is the one that costs a user effort: by far the smallest, and the hardest to inflate.
🔍 Click to zoom - three true MAUs for one product; the number to quote is the one built on the action that means value, and the tail is where automation hides
LiveThree definitions, one product, one month2 min▶

Any event. Starring a repository, commenting, pushing, anything at all: MAU 79,555, DAU/MAU 6.4 percent. This is the widest possible definition and the one most dashboards default to, because it is the easiest to compute and the largest number.

Pushed code. Only accounts that pushed at least once in a sampled hour: MAU 48,946, DAU/MAU 6.6 percent. Far narrower, and notice the ratio barely moved. The stickiness of the people who push is about the same as the stickiness of everyone; what changed is how many people there are.

Opened an issue or pull request. Accounts that did the thing that starts collaboration: MAU 8,993, DAU/MAU 5.0 percent. A fraction of the first number. If GitHub's product review quoted "monthly actives" without the sentence underneath, three honest analysts could bring three numbers between 8,993 and 79,555 and each would be right.

The lesson is not that the small number is correct. It is that the definition is the metric. Pick the action that means a user got value, write it down, and quote nothing that is not built on it.

LiveVanity versus signal2 min▶

The test. Could the team raise this number next quarter without a single user getting more value? If yes, it is vanity. "Any event" fails the test: a notification campaign that brings people back to glance at a page raises it, so does a bot, so does a new client that fires an event on open. "Opened an issue or PR" mostly passes: it takes a human intention, it is hard to automate without it showing, and it is rare enough that a real change is visible.

Vanity is not useless. The wide number tells you about reach; it just cannot tell you about value, and the failure mode is quoting one when you mean the other. A defensible review shows both, labelled, and never lets the wide one stand alone.

Self-studyWhere automation hides2 min▶

The engagement sample drops the 98 accounts whose login ends in [bot], because GitHub labels those. It cannot drop the rest. Among the 105 accounts seen on 22 or more days, a crude name check (does the login contain bot, -ci, ci-, sync, deploy, commenter or assistant) flags 17. The check is deliberately crude and the page says so: it will miss automation with an ordinary name and it will flag a human called "deploy-dave". The point is not the 17. It is that the far right of a power-user curve, the bar a growth team most wants to see, is exactly where scheduled software looks like a devoted person, and any right-tail celebration needs a bot pass first. Session 2 of the analyst track met the same problem inside one hour, where the single busiest login, with 25,458 events, had no [bot] suffix at all.

Real world

The number that fell from 79,555 to 8,993 without anyone doing anything. The three MAUs above are the same accounts in the same 28 days. Nobody shipped a feature, nobody ran a campaign, and "monthly active users" went from 79,555 to 8,993 by changing one clause. If your product's engagement definition lives only in the analyst's query, it can be changed by accident, by a dashboard migration, or by whoever is asked to make the chart look better before the board meeting. It belongs on a page with a date and an owner, which is what you build next.

Build-along

Your engagement definition card

★ 22 min · everyone builds

No tools and no data. One page, four fields, and the tracking plan from session 1 open beside you. The card is the sentence under every engagement chart your team will ever show, written once, owned by someone, with a date. Session 4 adds the feature funnels on top of it and session 6 reads a whole dashboard through it.

VALUE ACTION ACTIVE MEANS PERIOD WHAT WOULD INFLATE IT Report Exported the moment a user takes something away 1+ Report Exported by a non-internal account, in the period, counted by id weekly (WAU) used on Mondays, not daily scheduled exports (bot) internal QA accounts a nudge email that opens it one row, one definition. The example row is constructed. Owner: growth PM · reviewed: date Any chart that says "active" points here. One action, one sentence, one period, one list. If the card needs a second row, you have two metrics.
🔍 Click to zoom - the example row is constructed; the four columns and the gold owner tab are the part to copy

Pick the one action that means a user got value. Open the tracking plan from session 1 and read the five names. One of them is the moment the product did its job: the report left the building, the message was read by someone else, the invoice went out. Not the login, not the page view, not the thing the user does on the way to value. If two candidates compete, choose the one a bot would find hardest to fake.

★ The choice, with the reason written downCandidates from the plan: Workspace Created · Invite Sent · Report Exported · Plan Upgraded · Integration Connected Rejected: Workspace Created (setup, happens once) · Plan Upgraded (rare, a billing event, not use) Chosen: Report Exported Because: it is the moment the product's output is used somewhere else; nothing upstream of it counts as value

Write the definition of active as one sentence. It names the action, the minimum count, who is excluded, the identity key and the window. Read it aloud. If a colleague could compute a different number from the same sentence, it is not finished.

★ Active, definedAn account is active in a week when it has 1 or more Report Exported events in that week, counted by account id (never by email), excluding accounts on the internal list and excluding exports fired by a schedule rather than a click.

Choose the period to match the product's natural frequency. If people would use it every day when everything is going well, the period is daily and the ratio is DAU/MAU. If it is a Monday product, a month-end product or a per-project product, choose weekly or monthly and say why. On the GitHub sample the daily ratio is 6.4 percent and the weekly one is 18.6 percent, on the same accounts and the same definition; the period alone nearly triples the headline. Pick it once and do not let it be picked again by whoever wants a bigger number.

★ The period, chosen on purposePeriod: weekly Ratio: WAU / MAU Because: reports are exported on Monday mornings for the weekly review; a daily ratio would read as failure on a healthy product

List what would inflate it without value. Bots and scheduled jobs, internal and test accounts, notification-driven opens, a client that fires the event on load, a migration that replays history. Every item gets a one-line answer: excluded, flagged, or accepted with a note. This list is what makes the metric defensible, because it is the list of questions you already answered before anyone asked.

★ Inflators, and what is done about eachScheduled exports excluded: property trigger = "schedule" Internal and QA accounts excluded: internal account list, reviewed monthly Nudge email opens accepted, but reported separately as "nudged actives" Event fired on page load flagged: engineering to confirm the trigger is the click A replay of history flagged: any backfill is announced before it runs
✗ The card that cannot be defendedActive = any event Period = whatever the dashboard defaults to Bots = we don't think we have many

Show the wide number too, labelled. A card with one strict definition does not forbid the reach metric; it names it. "Accounts with any event: 79,555. Accounts that opened work: 8,993." Two numbers with two labels are defensible. One number with no label is a question waiting to be asked in front of the board.

Homework

Before session 4

30 min
  1. Find the sentence under your engagement chart. Ask the analyst which event, which count, which identity key and which window define "active" in the number the leadership team sees. If the answer is "the tool's default", write that down: it is the finding.
  2. Ask for the histogram. Days active per user for the last full month, as bars, not as a ratio. Look at the left bar first and the right bar second, and write one sentence about the shape before anyone tells you what it should be.
  3. Run the crude bot check on the right tail. Take the accounts active on 22 or more days and read their names. Count the ones that look like scheduled software or internal accounts. Ask what share of the right bar they are; if nobody knows, that is a second finding.
  4. Bring the card. Session 4 puts feature adoption on top of your value action, and its first question is which of your features are used again after the first try; the card decides what "used" means.
Check yourself

Three questions

Clear all three and leader session 3 gets stamped in your passport.

1 · Two products report the same DAU/MAU. What can you conclude about their users?

Jin and Chen's whole argument for the power-user curve is that a ratio is a mean of a mixed population. The histogram of days active is the only view that shows whether there is a core at all.

2 · The GitHub L28 on this page leans hard left: 66.7 percent of accounts were seen on one day in 28. What is the honest reading?

A defensible chart carries its sampling and its bias direction in the footnote. This one is sampled at 03:00 and 15:00 UTC on 5 percent of accounts, so it understates the right tail; saying so is what makes it usable.

3 · On the same 79,555 accounts, MAU is 79,555 under "any event", 48,946 under "pushed code" and 8,993 under "opened an issue or PR". Which number should the product review quote?

The definition is the metric. Neither size is a virtue; what is defensible is a definition chosen for value, written down, and quoted with its label so the reach number cannot be mistaken for it.

Sources covered

What this session teaches, and where it came from

✓DAU/MAU as a single number that "blurs this variance"; the power-user curve; "L30 (coined by the Facebook growth team)"Jin, L. and Chen, A. (2018). "The Power User Curve: The best way to understand your most engaged users." a16z, 6 August 2018; read at source
◐Facebook "60%+ of its MAUs coming back daily"Their claim in the same post, not verifiable from outside; taught as an illustration of the smile shape, never as a benchmark
✓The GitHub L28, the three MAUs, DAU/MAU and DAU/WAU, the 62 all-28-day accounts, the 105 on 22+ days and the 17 bot-like namesGH Archive. https://www.gharchive.org/ (hourly archives, data.gharchive.org); 5 percent of accounts, 03:00 and 15:00 UTC, 2024-03-04 to 2024-03-31; measured in this build by fetch-gharchive-actors.py and engagement-numbers.py
◐The sampling caveatTwo hours a day cannot see daily users active at other hours; the curve understates the right tail by an amount the sample cannot compute, and is never presented as GitHub's true stickiness
✓Vanity versus signal, the definition of active as the metricStandard practice; every count on this page from the same 79,555 accounts under three written definitions
◐The Report Exported card, the stickiness slide storyConstructed for the build-along and labelled a common pattern; not from any real product, no numbers

Every fact on this page, and its verification tier, is recorded in the course's source map.

Leader session 3 cheat sheet · pin this

A ratio blurs the shapeDAU/MAU is a mean; two different populations can print the same number.
Draw the power-user curveDays active per user, as a histogram. L30, or L28 for four weeks. Look for a right bar.
Left-leaning is normal, say whyGitHub sample: 66.7% on one day. Part real, part two-hour sampling. Footnote both.
The definition is the metricSame accounts: MAU 79,555 / 48,946 / 8,993 by what counted as active.
Vanity testCan the team raise it without a user getting value? Then label it reach, not engagement.
Check the right tail105 accounts on 22+ days, 17 bot-like names. Streaks mark cron jobs as well as fans.