Where we start
Session 3 settled what "active" means. Tonight's question is earlier in a life: is there something a new project does in its first hours or days that separates the ones that come back from the ones that do not? GH Archive cannot see sign-ups, so the unit is a birth it can see: a public repository created in one hour, 2024-03-04 15:00 UTC, the first new repository of each non-bot account. 8,348 of them. Their first seven days were sampled every second hour, and "came back" means at least one non-bot event in the sampled hours of days 28 to 41, weeks five and six. 325 of 8,348 came back, 3.9 percent. Most new projects are never touched again, and that is the first finding before any definition is written.
★ What you walk out with tonight. A candidate matrix of six behaviours by three windows, one function that scores any cell of it against the same outcome, the eight preset rows the bench in session 8 uses, and a sweep loop you run yourself. Plus the sentence that keeps the whole exercise honest: "seen on two days" predicts return at 5.44x because it is a small copy of return, not because it can be assigned.
Write down every early behaviour you can count
5 min liveA hunt needs a list. Six behaviours can be counted for every repository in its early life: events of any kind, pushes of code, separate days seen, issues or pull requests opened, different people active, and stars received. Three windows: the birth hour, the first day, the first seven days. That is eighteen cells, each of which becomes a candidate once you pick a threshold, and every one of them is scored against the same outcome, came back in weeks five and six. Eight cells have been scored and checked twice, once in Python and once in the browser engine, and they agree on every count. The rest are yours to run tonight.
LiveThe outcome comes first, and it is fixed2 min▶
Every candidate is scored against one outcome that never moves: a non-bot event on the repository in the sampled hours of days 28 to 41. Fixing it before the hunt is what stops the hunt from cheating. If you let the outcome drift with each candidate, "pushed on day one" gets scored against "pushed again in week five" and "opened an issue" against "issues in week five", and every candidate looks like the one. One outcome, one denominator of 8,348, one count of 325 returners, and every definition has to earn its lift against the same bar.
The 3.9 percent is low and that is expected: public repositories are cheap to create and most are a test, a fork of a tutorial or a one-off script. A workspace-based product would see a higher base rate, but the shape of the hunt is identical: a birth event the stream can see, a short early window, a later window that says "still here".
Self-studyWhat the six behaviours can and cannot see3 min▶
Each count is "seen in the sampled hours": every second hour of the first seven days, 84 files, then every second hour of days 28 to 41, 168 files. A push at 15:30 in a sampled hour is seen; a push at 16:30 is not. That undercounts every behaviour equally, so lifts between candidates stay comparable while every absolute count is a floor. Events by [bot] logins are dropped everywhere; bots without the suffix cannot be told apart from people by name and stay in.
"Separate days seen" is only meaningful in the seven-day window: in the birth hour and the first day it is 1 for every repository that has any event at all, so those two cells are not candidates and the grid says so. Stars are the odd one out in the list, and Part 3 says why.
Slack's 2,000 messages. Stewart Butterfield told First Round Review in 2015 that a team which has exchanged 2,000 messages has "really tried" Slack, and that 93 percent of such teams were still using it. Read the sentence the way tonight's grid reads a cell: the threshold was chosen "based on experience of which companies stuck with us and which didn't", and a team that reaches 2,000 messages has, by definition, already stayed long enough to send them. It is a real finding about the teams that kept going and a weak one about what makes a new team keep going. The 93 percent is the return rate of the activated; the number the interview does not give is the return rate of the rest, which is what a lift needs.
Three numbers per candidate, and none of them is enough alone
6 min liveTake "pushed code on the first day". 2,779 repositories did, 33.3 percent of the cohort. Of those, 151 came back, 5.4 percent. Of the other 5,569, 174 came back, 3.1 percent. Lift is the ratio, 1.74x. The difference is 2.3 points. And of all 325 returners, this definition caught 151, 46.5 percent. Move the same behaviour to a seven-day window and every number changes: 3,268 activated, 6.4 percent came back against 2.3 percent, lift 2.84x, 64.6 percent of returners caught. Same behaviour, different threshold, a different story. That is why a single hand-picked definition is a guess, and a sweep is the job.
LiveReading four canon rows4 min▶
push1, pushed on day one. 2,779 activated, 33.3 percent. 151 came back, 5.4 percent, against 174 of 5,569, 3.1 percent. Lift 1.74x, difference 2.3 points, 46.5 percent of returners caught. The definition most teams write first, and it is barely better than the base rate.
push7, pushed in week one. 3,268 activated, 39.1 percent. 210 came back, 6.4 percent, against 115 of 5,080, 2.3 percent. Lift 2.84x, difference 4.2 points, 64.6 percent caught. A wider window catches more returners and separates them better, because a push on day four is a return before the return window opens.
days2, seen on two or more days in week one. 1,186 activated, 14.2 percent. 154 came back, 13.0 percent, against 171 of 7,162, 2.4 percent. Lift 5.44x, difference 10.6 points, 47.4 percent caught. Far better per head than either push rule, at the cost of missing half the returners.
ppl2, a second person in week one. 288 activated, 3.4 percent, a small group. 53 came back, 18.4 percent, against 272 of 8,060, 3.4 percent. Lift 5.45x, difference 15.0 points, 16.3 percent caught. The highest return rate in the table and the smallest reach; with 288 rows the lift is real but the second decimal is not worth arguing over.
Self-studyWhy a sweep, not a pick3 min▶
The three numbers pull in different directions. Raising a threshold makes the activated group smaller and usually purer, so lift rises and returners caught falls. Lowering it does the opposite. "Three or more events in week one" calls 46.0 percent of the cohort activated and catches 71.4 percent of returners at 2.93x; "seen on two days" calls 14.2 percent and catches 47.4 percent at 5.44x. Neither is wrong. A definition meant to trigger an onboarding nudge wants reach, so the broad one; a definition meant to flag which new projects are worth a human's attention wants purity, so the narrow one.
The sweep also shows you what the data does not do. The brief for the bench expected "total events" to reverse at every window, a pure volume trap. The rows do not agree: volume reverses only in the birth hour. Ten or more events in the birth hour, 117 repositories, has a lift of 0.88x with 4 returners, while 2 or more in the same hour is 1.49x and 5 or more is 1.58x. A sweep prints what is there, and what is there is the lesson.
Facebook's 7 friends in 10 days, and what it hides. The famous line is reported: a transcript of Chamath Palihapitiya's 2012 talk has him saying the biggest thing the growth team realised was to get any individual to 7 friends in 10 days. Nothing about the analysis behind it was ever published. Andrew Chen is reported, via Mixpanel's write-up, to have said the number could as easily have been 10 in 12 or 5 in 1, and Benn Stancil at Mode called it a rally cry rather than a threshold. Tonight's scatter is the reason those caveats matter: on this cohort, the neighbouring thresholds of one behaviour sit at 1.74x and 2.84x. A single cell quoted without its neighbours is a slogan, and a slogan is not a definition.
The best predictor is usually the outcome arriving early
4 min liveLook at what won. "Seen on two or more days" is 5.44x because being seen on a second day in week one is a small return, measured before the return window opens. "A second person showed up" is 5.45x because a project two people touch is already a different kind of project. "Received a star" is 3.35x and the owner did not do it; someone else did. None of these is a button. A lift tells you which early rows look like the late rows. It does not tell you that making a quiet project look like a busy one will make it come back.
LiveThree ways a strong predictor fools you3 min▶
It is the outcome, early. Coming back on a second day and coming back in week five are the same behaviour at two distances. Of course they correlate. The right reading is "the projects that will return start returning immediately", which is useful for forecasting and useless as an instruction.
Someone else did it. A star is given by another account. A second person showing up is, by definition, not the owner. Both say "this project matters to more than one person", a property of the project, not an action you can ask the owner to take.
The group is small. 288 repositories for ppl2, 214 for work1, 194 for star1, 117 for burst10. The bench's break button shuffles each definition across the cohort with a fixed seed so the group keeps its size and loses its link to the repository; the shuffled lifts land between 0.85x and 1.33x, and the small groups wander furthest from 1. A 3.35x on 194 rows is a real signal; a 1.3x on 117 rows is indistinguishable from nothing.
To move from "predicts" to "causes", assign the behaviour at random and measure return in both arms. That is the whole topic of Experimentation, and it is the only route.
Self-studyWhat you can honestly do with a predictor2 min▶
A predictor without a cause is still worth having. It scores new projects on day seven so a team can look at the ones likely to stay, it gives a leading indicator that moves weeks before the retention number does, and it names a candidate for the experiment. What it cannot do is set the target: "get 40 percent of new projects to a second day" turns a measurement into a quota, and a quota changes the behaviour it measures. Write the definition down with its lift, its reach and its group size, and label it a signal until an experiment says otherwise.
The busy birth hour. Ten or more events in the first hour looks like enthusiasm. In this cohort it is 117 repositories, 4 of which came back, lift 0.88x. The count is noise-sized, so the honest claim is "no signal", never "harm". What the row is showing is setup: a template pushed, branches created, a workflow firing, a lot of activity that is all one act. A common pattern in workspace products is the same: an import that creates fifty objects in a minute reads as a power user on any volume metric and is one click. Setup activity is not engagement, and a sweep that includes the birth hour is how you find out where your product's setup burst ends.
Score the cohort, reproduce the canon, then sweep
★ 22 min · everyone buildsStandard-library Python and one file, the same cohort the bench in session 8 runs in the browser. The file is real GH Archive rows, one entry per repository, already reduced to the six behaviours in three windows plus the outcome; it is 285,995 bytes. Every number in the comments below is what this code prints. The sweep at the end prints far more rows than fit on a page, so it carries no comments: compare your printout with the bench.
Download the cohort and strip the wrapper. The file is JavaScript that assigns one object to window.PA_SAMPLE. Cut the assignment and the trailing semicolon and what is left is JSON.
Write evaluate. One function scores any cell of the grid at any threshold: flag the rows at or above k, count who came back on each side, divide. "Separate days" only exists as an array for the seven-day window; in the shorter windows it is 1 for any row with an event, which the helper derives so the sweep can still ask.
Print the eight canon rows. These are the bench presets, and the comments are what the code prints. The same eight counts come out of the browser engine.
Check the birth-hour neighbours. The two thresholds either side of burst10 show that volume reverses only at the top of the birth hour, not across it.
Sweep every cell. Six behaviours, three windows, thresholds 1 to 10, skipping a cell once nobody is left in it. This prints well over a hundred rows and none of them is quoted here: compare your printout with the bench, and bring the row you would defend to session 8.
Print the group size next to every lift. A sweep hands you a hundred lifts and the highest one is nearly always on the smallest group. The bench flags groups under 300 as small; do the same in your own printout, and read the shuffled band in Part 2 before you get excited about a 1.3x.
Before session 5
30 min- Pick the row you would defend. From your sweep, choose one definition for "this new project is worth a human's attention" and one for "this new project should get the onboarding nudge". Write each as a sentence with its activated share, lift, returners caught and group size. Bring both to session 8; the bench will score them.
- Find the setup burst. In the birth-hour column, print lift for every threshold from 1 to 10 of events of any kind and write down where it stops rising. That is where setup ends in this cohort; describe what the same boundary would look like in a product you know.
- Argue the other side. For your strongest row, write three sentences: one explaining it as the outcome arriving early, one as something someone other than the owner did, one as a real lever. Say which you believe and what experiment would settle it.
- Keep the file. Session 5 goes back to accounts rather than repositories and builds the power-user curve from real rows; the same standard-library pattern loads that data too.
Three questions
Clear all three and analyst session 4 gets stamped in your passport.
1 · "Pushed code in week one" catches 64.6 percent of returners at 2.84x; "seen on two or more days" catches 47.4 percent at 5.44x. Which is the better activation definition?
Lift, difference and returners caught pull in different directions, and a threshold moves all three at once. The sweep exists so the choice is made with all three numbers in view and named for the decision it serves.
2 · "Seen on two or more days in week one" has a lift of 5.44x on this cohort. What does that support?
A lift compares rows that already differed. Coming back on day two and coming back in week five are the same behaviour at two distances; proving that pushing people into the first changes the second needs random assignment.
3 · Ten or more events in the birth hour: 117 repositories, 4 came back, lift 0.88x. Its shuffled version reads 1.32x. What is the honest sentence?
The shuffle keeps the group size and removes the link, and small groups land furthest from 1.0x. A 0.88x and a 1.32x on 117 rows are the same finding: nothing measurable. "Harm" would need a group large enough to be sure, and this one is not.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Analyst session 4 cheat sheet · pin this
Tonight scored projects in their first week. Session 5 goes back to people and asks how often they show up: the power-user curve built from real rows, a 5 percent sample of GitHub accounts across 28 days, where 66.7 percent of active accounts were seen on a single day, DAU/MAU lands at 6.4 percent, and the right tail of the histogram turns out to be partly automation that no bot suffix admits to.