Where we start
A bench is a fixed set of rows and a fixed question, so that when two definitions give two answers the only thing that differed was the definition. The rows are real: every public repository whose creation event appears in GH Archive between 15:00 and 15:59 UTC on 4 March 2024, one per account, bots excluded. The question is the one activation is supposed to answer: does doing this early predict coming back later? Every number on the bench is a count. Nothing is modelled, so there is no tag to read before the number except one: the counts are of what the public stream saw in the hours that were sampled.
★ What you walk out with tonight. Measured lifts for eight activation definitions on real rows, the habit of reading "share activated" and "returners caught" beside the lift, a shuffled floor to judge every lift against, and the honest sentence a strong correlation earns, which is not "this causes staying".
A birth the diary can see
6 min liveActivation is measured from a start. GH Archive cannot see an account being created, and "the first time an account appears in the window" mostly finds old accounts that were quiet before it opened, so that heuristic is out. What it does see, exactly once, is a repository being born: a CreateEvent whose payload says repository. So the unit is a new repository, the way a SaaS product counts a new workspace.
LiveFour decisions, stated once3 min▶
The unit is a new public repository, because its birth is the only start the public stream records. The bench says projects, never people, and only public ones.
One repository per account, bots out. Accounts whose login ends in [bot] made 127 of the hour's 10,384 repositories, and a few human-named accounts made dozens each. Keeping each account's first repository gives 8,348 independent starts. Bot events are dropped from every count. Automation without the suffix cannot be recognised from a name and stays in; session 5 shows how much of it there is.
Came back means weeks 5 and 6. Any event by a non-bot account on the repository in the sampled hours of days 28 to 41. 325 of 8,348 did, 3.9 percent. That low base rate is a finding, not a flaw: most new repositories are a one-afternoon experiment.
Every second hour is sampled. 252 hourly files, about 37 GB, streamed and filtered down to the cohort's 44,573 events. A repository active only in odd hours is invisible, at both ends equally.
Self-studyWhat the bench is not2 min▶
It is not a retention analysis: there is one outcome window, not a curve, and cohort curves belong to the customer retention course. It is not an experiment: nobody was nudged toward any behaviour, so no lift here says what would happen if you pushed users to do the thing. And it is not GitHub's own activation metric, which is not public. It is a fixed, real cohort on which definitions can be compared honestly, and that is exactly the job of a bench.
Eight definitions, computed in front of you
★ 26 min · the main eventWork down the eight in order, then write your own in the row under them. For each, read the verdict, then the five readouts. Say out loud how many repositories the definition calls activated before you look at the lift, because a lift on 3 percent of the cohort and a lift on 40 percent are different claims. The last two are the volume trap. After all eight, press the break button on two of them.
Read it in this order. Activated share, then the two came-back rates with their counts, then the lift, then returners caught. The first definition is the one most teams write first, one push on day one, and it is worth sitting with its 1.74x before moving on.
Same repositories, eight answers
6 min liveThe bench above computes these live; the numbers here are the ones it prints on the 8,348 rows, written down so they can be argued with after class. An independent Python run reproduces every count exactly.
| Definition | Activated | Came back, activated | Came back, the rest | Lift | Difference | Returners caught |
|---|---|---|---|---|---|---|
| Pushed code on day one | 2,779 · 33.3% | 151 · 5.4% | 174 of 5,569 · 3.1% | 1.74x | +2.3 pts | 46.5% |
| Pushed code in week one | 3,268 · 39.1% | 210 · 6.4% | 115 of 5,080 · 2.3% | 2.84x | +4.2 pts | 64.6% |
| Came back on a second day | 1,186 · 14.2% | 154 · 13.0% | 171 of 7,162 · 2.4% | 5.44x | +10.6 pts | 47.4% |
| A second person showed up | 288 · 3.4% | 53 · 18.4% | 272 of 8,060 · 3.4% | 5.45x | +15.0 pts | 16.3% |
| Opened an issue or pull request | 214 · 2.6% | 31 · 14.5% | 294 of 8,134 · 3.6% | 4.01x | +10.9 pts | 9.5% |
| Got a star | 194 · 2.3% | 24 · 12.4% | 301 of 8,154 · 3.7% | 3.35x | +8.7 pts | 7.4% |
| 3 or more events in week one | 3,839 · 46.0% | 232 · 6.0% | 93 of 4,509 · 2.1% | 2.93x | +4.0 pts | 71.4% |
| 10 or more events in the birth hour | 117 · 1.4% | 4 · 3.4% | 321 of 8,231 · 3.9% | 0.88x | -0.5 pts | 1.2% |
Every cell is a count or a ratio of counts from the 8,348 rows. Python and the in-browser engine agree on every count.
LiveFinding 1: the definition most teams write first is the weakest real one2 min▶
One push on day one calls a third of the cohort activated, and those repositories came back at 5.4 percent against 3.1 percent: a lift of 1.74x. Giving the same behaviour a week raises it to 2.84x, because most repositories that are going to be worked on again get pushed to again. Coming back on a second day, a much narrower rule, reaches 5.44x on 14.2 percent. The behaviour a team can see in the onboarding funnel is not the behaviour that separates projects that live from projects that do not.
LiveFinding 2: breadth and lift trade against each other, so read both2 min▶
A second person showing up has the same lift as a second day, 5.45x against 5.44x, on a quarter of the size: 288 repositories rather than 1,186, and it catches 16.3 percent of everyone who came back rather than 47.4. An activation metric the team will steer by needs enough of the cohort in it to move. The last column is the one that says whether a definition describes the product or a niche of it.
Self-studyFinding 3: the strongest predictors are the least like levers2 min▶
Coming back on a second day is an early version of the outcome itself: a repository that is worked on twice in week one is, unsurprisingly, likely to be worked on in week five. A star is something other people do. Neither is a button a product team can press. That is not a reason to reject them as activation metrics; a leading indicator is useful for knowing early whether a cohort is healthy. It is a reason never to present one as the cause of staying. Proving that pushing users toward a behaviour changes their return needs an experiment, which the experimentation course covers.
Volume is broad, and a shuffle is the floor
5 min live"Count everything, set a bar" is the default activation metric of many dashboards. On this cohort it is not useless, but it is broad, and in its sharpest form, a busy first hour, it carries no signal at all. The break button shows what "no signal" looks like at every group size, which is what every real lift should be judged against.
LiveThe volume trap, stated with its nuance2 min▶
"3 or more events in week one" calls 46.0 percent of the cohort activated and catches 71.4 percent of everyone who came back, at a lift of 2.93x. It is not worthless: busier repositories do come back more. But it separates little more than half as well per head as "came back on a second day", and it activates so many repositories that almost any onboarding change will move it. That is what makes volume attractive on a dashboard, and what makes it a weak steering metric.
The sharpest form is a busy birth hour. Ten or more events in the first hour is setup activity: creating the repository, the default branch, a flurry of initial pushes. 117 repositories qualify and 4 came back, 3.4 percent against 3.9 for the rest: a lift of 0.88x. Four is a noise-sized count, so the honest reading is no signal, not harm. The shuffled version of the same definition reads 1.32x, which is the size of the wobble a group of 117 has by chance alone.
LiveWhat the break button proves2 min▶
The shuffle keeps the number of activated repositories exactly the same and deals those activated slots to random repositories, with a fixed seed so everyone sees the same result. Any link between the behaviour and the repository is gone; only the group size remains. Across the eight definitions the shuffled lift lands between 0.85 and 1.33 and the difference between -0.6 and +1.3 points. Small groups wander furthest: the three smallest, 117 to 214 repositories, give the three shuffled lifts furthest from 1.
That band is the practical floor. A definition whose real lift sits inside it has not shown it is more than chance on a cohort this size; one whose real lift sits at 5x has. It is a quick check, not a significance test, and it is honest about what it does: it asks whether the bench is measuring the behaviour or measuring an accident of group size.
The onboarding checklist that raised activation and nothing else. A team defines activation as "completed 5 onboarding steps", ships a checklist that nudges users through them, and watches activation jump. Retention does not move. The five steps were setup activity, the equivalent of the busy birth hour: a definition whose volume the product can raise at will and whose link to staying was never tested. The bench's move is the fix: compare the definition's lift against its shuffled floor before building a roadmap on it, and prove causation with an experiment after.
Before the product review
30 min- Find a definition that beats 5.44x with your own row on the bench, and write down its activated share and returners caught next to the lift. Decide whether you would steer a team by it.
- Break your best definition and record its shuffled lift. Is the real lift outside the 0.85 to 1.33 band?
- Write the honest sentence for your product's current activation metric in the form the bench uses: "X percent are activated; they come back at A against B; lift L; it catches C of returners; nobody has tested whether it causes anything."
- Take that sentence to leader session 6, the product review, where it is the first slide.
Three questions
Clear all three and analyst session 8 gets stamped in your passport.
1 · Why does the bench use new repositories instead of accounts first seen in the window?
A cohort must start at a moment the data actually records. When the birth you want is invisible, use a birth that is visible and say plainly what it stands for: here, projects rather than people.
2 · "3 or more events in week one" catches 71.4 percent of returners at 2.93x. "Came back on a second day" catches 47.4 percent at 5.44x. What is the fair reading?
A lift is a claim about a group of a certain size. The three readouts together say whether a definition describes the product or a niche, and whether it separates or merely counts.
3 · After shuffling, the eight lifts fall between 0.85 and 1.33. The busy birth hour's real lift is 0.88x on 117 repositories. What should you conclude?
An anti-lever must carry its honest nuance. The shuffle shows how far a group of 117 wanders by chance, and the real value sits inside that wander. The claim it earns is "no signal", which is already enough to keep it off a roadmap.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Analyst session 8 cheat sheet · pin this
That is the end of the analyst track. The natural next step is leader session 6, the product review, which takes the honest sentence from your homework and asks the six questions every product dashboard should survive: which definition, which cohort, which denominator, what the diary cannot see, where the bots are, and what has actually been shown to cause what.