The product review

1 · Which definition? The same 28 days of sampled GitHub accounts give a monthly active count of 79,555 if any event counts, 48,946 if only a push counts, 8,993 if only opening an issue or pull request counts. A tile that says "MAU" without saying which one is three different claims.

2 · Which cohort, and when does it start? A cohort has to begin at a moment the diary recorded. The first time an account appears in a window is not its first day; the bench's cohort is 8,348 repositories born in one hour, one per non-bot account, because a repository's birth is written down and an account's is not.

3 · Which denominator? A lift is a ratio of two return rates; it says nothing about how many people it applies to. "3+ events in week one" calls 46.0 percent activated at 2.93x; "seen on 2+ days" calls 14.2 percent activated at 5.44x. Read the share beside the lift, always.

4 · What can the diary not see? The engagement numbers come from two sampled hours a day, and the activation bench from every second hour. Every count is "seen in the sampled hours", and a daily user who works at other times looks occasional.

5 · Where are the bots? In one hour, 409 [bot] logins produce 15.5 percent of events, and the busiest account of all, censameesss with 25,458 events, carries no suffix. Filtering by name catches the honest bots only.

6 · What has been shown to cause what? Usually nothing. A lift is a correlation. Proving that pushing people toward a behaviour changes whether they stay needs an experiment, which is its own discipline: learn experimentation with phoebe.

The questions are ordered so that each one only makes sense once the one before it is answered. A denominator cannot be checked until the cohort is fixed, and the cohort cannot be fixed until the definition is written. Sampling and bots come next because they change the counts inside an agreed frame. Cause comes last because it is the only question that the dashboard, on its own, can never answer.

Asking in order also keeps the meeting civil. "Which definition is this?" is a question nobody is embarrassed by. Starting at question 6, "is this causal?", sounds like an accusation, and the review turns into a defence of the chart rather than a reading of it.

Definition. Written down, and narrow enough to check: a repository counts as activated if it is seen on two or more separate days in its first seven, and it "came back" if any non-bot event touches it in days 28 to 41, weeks 5 and 6. Pass.

Cohort. 8,348 repositories, each the first new repository of a non-bot account in the hour 2024-03-04 15:00 UTC. The start is a recorded birth, not a first sighting. The note to keep: these are public projects, not people, the way a SaaS product counts new workspaces.

Denominator. 1,186 repositories, 14.2 percent, are activated. They came back at 13.0 percent; the other 7,162 came back at 2.4 percent. Across the whole cohort only 325 came back, 3.9 percent, so the definition catches 47.4 percent of returners. Set beside "3+ events", which calls 46.0 percent activated at 2.93x and catches 71.4 percent of returners, it is a sharper rule that covers fewer. Both facts belong on the slide.

Sampling. The bench read every second hour: 84 sampled hours for days 0 to 6 and 168 for the return window. "Seen on 2+ days" means seen in the sampled hours of two days. A repository worked on only in the unsampled hours looks inactive. The effect on the lift is unknown, so the caveat stays on the slide rather than being argued away.

Bots. Events by [bot] logins are dropped everywhere, and 127 of the hour's 10,384 new repositories came from them. Automation without the suffix cannot be detected reliably from a name and stays in. One account created 115 repositories in the hour; the cohort keeps only its first. Say both things out loud.

"Activates" is a causal verb. It invites a roadmap: nudge new projects toward a second day and more of them will stay. The bench cannot support that. Coming back on a second day is an early version of the outcome itself, coming back later. A repository its owner already cares about will show up on day two and on day thirty for the same reason.

The shuffle is the other half of the reading. When the bench keeps the group size at 1,186 and randomly reassigns which repositories are "activated", the lift falls to 0.85x. That is what no link looks like at this size, and 5.44x is far from it. So the association is real. What it is not, yet, is a lever. The rewritten sentence at the bottom of the figure keeps every number and changes one clause.

The five artifacts from sessions 1 to 5 are the review's reference shelf. The tracking plan answers question 1 and holds the blank-pages tab for question 4. The activation candidate with its honest sentence answers questions 2 and 3. The engagement curve shows which definition of active was used and what sampling it rests on. The adoption table keeps breadth, depth and time apart, so a setup artefact like "branch created", adopted by 68.8 percent, is not mistaken for value. The AI metric sheet says which of acceptance, persistence or characters kept was chosen, and for which decision.

Ask for these before the meeting, not during it. A number whose definition has to be looked up live is a number that will be decided on before the answer comes back.

Every headline arrives in one fixed shape, the same one the analyst track's bench uses: "X percent are activated; they come back at A against B; lift L; it catches C of returners; nobody has tested whether it causes anything." The shape does the reviewing for you. It cannot be filled in without a definition, a share, both return rates and a coverage figure, and its last clause is a standing reminder that a correlation has not yet become a lever.

Filled in for the busiest-birth-hour rule, it reads: 1.4 percent are activated; they come back at 3.4 percent against 3.9; lift 0.88x; it catches 1.2 percent of returners. With only 117 repositories in the group, and a shuffled version of the same size reading 1.32x, the honest verdict is "no signal", not "harm". The sentence keeps the reader from over-reading in either direction.

Send it back, politely and with the question written down, when any of these is true:

• No written definition for a metric on the slide, or two teams using the same name for different counts.
• A cohort that starts at "first seen" in a source that never recorded the real start.
• A lift with no share beside it, or no base rate for the whole group.
• A small group, under about 300, presented without a shuffled or otherwise "no signal" comparison. On the bench, groups of 117 to 288 wander furthest from 1 when the link is removed.
• No statement about bots or blind spots in a stream known to carry both.
• A causal verb ("drives", "activates", "makes users stay") for something nobody has tested.

Sending back is not rejection. The finding may be exactly right; it is not yet in a form a decision can rest on. Reading metrics as an executive more generally is its own subject: learn data literacy with phoebe.