learn-people-analytics-with-phoebe / Deep-dive session 2 of 10
Learn People Analytics with Phoebe · Deep-dive session 2 of 10

Attrition as survival

"How long do people stay?" is not a question about an average. Most tenures in any staff table have not finished: the person is still employed, and all the table knows is that they have stayed at least this long. That is censoring, and it is the reason tenure is a time-to-event problem with its own arithmetic. Tonight fits a Kaplan-Meier survival curve by hand in pandas to the constructed Apex table, reads 0.940 at one year and 0.778 at five, and then does the more useful thing: proves that every one of those numbers is too high, because the table is a photograph of survivors and nobody who left before 2026 is in it. The direction of the bias is certain. Its size is not, and the honest report says both.

🟡 Core HR analysts, people data teams, anyone asked "what is our average tenure" Python, pandas 45 min
0-3 · Welcome 3-18 · Censoring, the staircase, the bias 18-40 · Build-along: Kaplan-Meier in pandas 40-45 · Q&A
Part 0

Where we start

Deep-dive session 1 ended with a bridge that reconciles and a table known to be a photograph: everyone employed on 1 January 2026, plus the 210 people who left between then and 17 June. Apex is constructed, nobody in it is real, and the shape is the point. Tonight asks that photograph a question about duration. It will answer, in numbers to three decimal places, and the numbers will be wrong in a direction that can be proved from the shape alone.

🎫 Clear all three quiz questions to stamp your passport Live = presented in the 45 minutes Self-study = the full depth, after class ★ Build-along = run the code yourself

★ What you walk out with tonight. Tenure as a duration with an event flag; the Kaplan-Meier product written as a loop you can read; a survival curve on Apex with its four values; the two-sentence proof that a survivor table biases the curve upward; and the specification of the event log that would fix it.

Part 1 · covers tenure as time-to-event and censoring

Tenure is a duration, and most of it has not finished

5 min live

Ask an HR export for average tenure and it will give you the mean of a column. That column mixes two different things: for the 210 leavers, the length of a stay that ended; for the 861 still employed, the length of a stay that is still running and will end at some unknown later date. Averaging them is averaging finished races with races in progress. The fix is not a better average; it is a different object, the survival function, which asks at every tenure t what share of people are still there.

SEVEN STAYS, THREE FATES: SEEN TO END, STILL RUNNING, NEVER RECORDED 2012 2016 2020 2024 1 JUL 2026 CALENDAR TIME left 2020: absent left 2023: absent open circle: censored at 1 July 2026, 861 rows, the stay is still running wine dot: an exit the table can see, 210 rows, all between January and June 2026 Faint stays ended before 2026 and are not rows at all. Constructed Apex table; the seven stays are drawn to show the three fates.
🔍 Click to zoom - the open circle is the whole subject: a stay that has not ended yet is information, not a gap
LiveThree kinds of row, and what each one knows3 min▶

Every row in Apex becomes two numbers: a duration t, and an event flag. For the 210 with a last day, t is the years from join to exit and the flag is 1: the stay ended and we saw it end. For the 861 without one, t is the years from join to the snapshot date, 1 July 2026, and the flag is 0: the stay is at least t long, and that is all we know. The second kind is censored. A censored row is not missing data; it is a lower bound, and the survival arithmetic uses it as one.

There is a third kind of row, and it is the one that matters most tonight: the row that does not exist. Anyone who joined Apex and left before 1 January 2026 is not in the table. Their stay ended, we would have seen it end, and we never will, because the export was taken after they had gone. The figure draws two of them faint. Deep-dive session 1 called the export a photograph; the survival curve is where the photograph starts to lie in a measurable direction.

The two medians already disagree. Among the 210 exits the table can see, tenure at exit has a median of 1.52 years: 29.5 percent left inside their first year and 61.0 percent inside two. Among the 861 still employed, tenure so far has a median of 5.01 years. Whatever the true curve looks like, the people who leave are leaving early, and the people who stay have stayed long. The curve has to reconcile both.

Self-studyWhy the mean of a tenure column is not a tenure2 min▶

Take the mean of Apex's tenure column and you get a number that goes up every month nobody leaves, because every censored stay grows by a month while the finished ones stay fixed. Take the mean of tenure at exit instead and you get 1.52 years as a median, which describes the leavers and says nothing about the 861 who have not left. Neither number is the average length of a stay at Apex. The survival function sidesteps the problem by never asking "how long" of any one person; it asks, at each t, "of the people who reached t, how many were seen to leave at t", and a censored person counts in the first group at every t up to their own and never in the second. The customer version of the same arithmetic, on subscriptions instead of staff, is in the Customer Retention course; the mechanics are identical and the data there has the shape this one lacks.

Real world

"Our average tenure is 5.0 years." A people team reports this from an export, and it is true of the people in the export on the day it was taken. It says the survivors have survived a while. It does not say that a person joining today should expect five years, and in Apex's constructed table it sits beside a median exit tenure of 1.52 years for the people who did leave. The two numbers describe two different populations, and the sentence "our average tenure is 5.0 years" quietly picks the flattering one. Constructed setting; the habit is common.

Part 2 · covers Kaplan-Meier by hand

The staircase: one step down per exit, sized by who was still at risk

5 min live

Kaplan and Meier's 1958 estimator is a product. Sort every stay by t. Walk up through them. At each exit, multiply the running survival by one minus one over the number of people still at risk just before it; at each censored stay, multiply by nothing and simply remove the person from the at-risk count. The result is a staircase that only steps down at an exit, with each step sized by how many people were still there to take it. On Apex the staircase reads 0.940 at one year, 0.872 at two, 0.813 at three and 0.778 at five.

THE STAIRCASE ONLY STEPS AT AN EXIT, AND THE STEP IS 1 / (PEOPLE STILL AT RISK) TENURE, YEARS S(t) 1.00 0.90 0.80 0.70 0 1 2 3 4 5 6 0.940 0.872 0.813 0.778 each step: S times (1 - 1/at risk) dashed: the true curve is lower; how much, this table cannot say Read at 1, 2, 3 and 5 years: 0.940 / 0.872 / 0.813 / 0.778. Year one alone holds 62 of the 210 visible exits. Constructed Apex table, 1,071 stays. Part 3 shows why every one of the four readings is too high.
🔍 Click to zoom - the step size is the whole method: an exit when 1,071 are at risk moves the curve less than an exit when 441 are
LiveThe product, in words, then in one loop3 min▶

Start with S = 1 and everyone at risk: 1,071. Take the shortest stay. If it ended in an exit, S becomes S times (1 - 1/1,071). Either way, one fewer person is at risk. Take the next shortest, and so on. By the end of the first year, the table has seen 62 exits, S has fallen to 0.940, and 946 people are still at risk. By two years: 128 exits seen, S 0.872, 803 at risk. By three: 180 exits, 0.813, 670 at risk. By five: 206 of the 210 exits have been used, S is 0.778, and 441 people remain at risk, all censored stays that will each remove one from the count and none from S.

Two properties follow from the arithmetic and are worth saying aloud. Censoring never moves the curve down, it only shrinks the denominator for the next exit, which is why a censored stay is information and not a gap. And the curve is only defined as far as the longest stay; Apex's longest is a little over fourteen years, so "median tenure" from this curve does not exist, because S never reaches 0.5.

The at-risk counts and exit counts above are computed from the constructed table in the build-along; they are the numbers the loop passes through on its way to the four readings.

Self-studyCox, named once, and where the customer version lives2 min▶

Kaplan-Meier describes one curve. The moment the question becomes "does Sales leave faster than R&D, adjusting for entity and join cohort", the tool is Cox's proportional hazards regression, "Regression Models and Life-Tables", Journal of the Royal Statistical Society Series B 34(2) 187-202, 1972. It models the hazard, the instantaneous rate of leaving at each t, as a baseline shared by everyone times a multiplier per covariate. It is named tonight and not fitted, for a reason that is the subject of Part 3: a regression on a biased curve inherits the bias, and a hazard ratio between two departments computed from a survivor table compares two distorted hazards. Fit it on an event log. The Customer Retention course fits the same curve to customers, where the event log usually exists, and is the place to see the estimator on data that can bear it.

Part 3 · covers the survivor-table bias

Why every number on the staircase is too high

5 min live

The estimator is fine. The input is not. Apex's rows are the people employed on 1 January 2026 plus six months of leavers, and that shape does something specific to the fraction inside every step. The proof takes two sentences and needs no number that is not already on the page.

WHO THE PHOTOGRAPH KEPT, AND WHO IT LOST IN THE TABLE: 1,071 rows 861 still employed at 1 July 2026 tenure so far: median 5.01 y, all of it in the at-risk count 210 exits, all between January and June 2026 tenure at exit, median 1.52 years NOT IN THE TABLE: left before 1 January 2026 count: unknown tenure at exit: unknown their exits: not in any numerator their years at risk: not in any denominator every stay that ended before 2026, gone exits seen at tenure t people at risk at tenure t the denominator counts every year the survivors spent at t, back to 2012 the numerator can only count exits between January and June 2026
🔍 Click to zoom - years of exposure that had no chance of producing a visible exit push every hazard down and every survival number up
LiveThe proof, in two sentences, and what it does not tell you3 min▶

Sentence one. At any tenure t, the estimator's denominator counts everyone in the table who reached t, and the 861 survivors reached t in 2015, 2019, 2023, whenever, so the denominator contains years of exposure from long before the export. Sentence two. The numerator at t counts exits at t, and the only exits in the table happened between January and June 2026, so every year of exposure before 2026 is in the denominator with no possibility of contributing to the numerator. A fraction whose denominator includes exposure that cannot produce an event is too small at every t; a hazard that is too small at every t gives a survival curve that is too high at every t. The direction is certain.

What the proof does not tell you is the size. The size depends on how many people left before 2026 and when, and those are the rows that are not there. The curve could be a little too high or a great deal too high; the table cannot distinguish the two. So the honest report of this curve is: "on the people present in 2026, 0.940 at one year, biased upward by an amount this table cannot measure". Both halves of that sentence are findings.

There is a hint of the size in a number already on the page. Half of the 210 visible exits happened inside 1.52 years, and 61.0 percent inside two, while the curve says 87.2 percent of stays survive two years. Those two facts can coexist only if a great many people who would have left early are not in the table to be counted. That is exactly what the photograph did.

Self-studyWhat an event log would fix, and what it would have to contain2 min▶

The fix is not a different estimator; Kaplan-Meier is doing its job. The fix is the input: one row per join and one row per exit, for everyone who ever joined, back to a start date that is stated. Then a person who joined in 2019 and left in 2022 is two rows, contributes three years to the denominator and one exit to the numerator, and the fraction at every t is exposure against events that could actually happen. The minimum event log for this session is four columns: person id, event date, event type (joined, left), and the reason if any. With it, the only censoring is the honest kind at the right edge, where stays are still running on the day of the export.

Two practical routes to that log. Ask the HR system for the change history of the status field rather than the employee export; most systems keep it and few are asked. Or, from today, keep every monthly export with its date, and diff consecutive months: ids that vanish left, ids that appear joined. Twelve months of that habit gives a real first-year hazard for the cohort that joined in the first month, which is one true number more than the photograph can ever give.

Real world

The board pack that said "first-year attrition is under 6 percent". The number came from a curve like tonight's, read at one year from an export like Apex's: 1 - 0.940. Nobody lied. The analyst had the estimator right and the table had quietly excluded everyone whose first year had already ended badly. When a cohort log was built from monthly exports the following year, the first-year figure for the tracked cohort was the first honest one the company had, and it was not under 6 percent. Constructed setting; the mechanism is the one drawn above.

Build-along

Kaplan-Meier in pandas, on 1,071 constructed stays

★ 22 min · everyone builds

Seven steps on materials/apex-staff.csv. No survival library: the estimator is one loop, and writing it once is the fastest way to understand what a library does with a censored row. Every printed number is what the code produces on the constructed table; the four readings match the ones in Part 2 exactly.

SEVEN STATIONS, AND THE INPUT THAT DECIDES WHAT THEY MEAN input A · Apex snapshot, biased input B · an event log, unbiased 1 · load 1,071 rows 210 last days 2 · tenure t years from join to exit or SNAP 3 · event flag 1 = exit seen 0 = censored 4 · sort by t at risk counts down from 1,071 5 · the product S *= 1 - 1/n per exit only 6 · read S(t) 0.940 · 0.872 0.813 · 0.778 7 · the bias direction: up size: unknown Stations 1 to 7 are the same code for both inputs. Only the input decides whether station 6 is a finding or a warning. Save the curve with the snapshot date and the words "survivor table" in the filename. Deep-dive session 8 plans from the honest version.
🔍 Click to zoom - station 7 is one printed sentence, and it is the only station a survivor table lets you get exactly right

Load, and name the snapshot date. The date the export was taken is the censoring date for every stay without a last day. It goes at the top, as a constant, with the window the table can see written beside it.

★ Step 1 · loadimport numpy as np, pandas as pd d = pd.read_csv("materials/apex-staff.csv", parse_dates=["Date Join", "Last Day"]) SNAP = pd.Timestamp("2026-07-01") # export date; censoring date for every open stay # the table can see exits from 2026-01-01 to 2026-06-17 and NO exit before that print(len(d), d["Last Day"].notna().sum(), d["Last Day"].isna().sum()) # 1071 210 861

Tenure as a duration, in years. Exit date if there is one, otherwise the snapshot date, minus the join date. Clip at zero so a join after the snapshot cannot produce a negative stay.

★ Step 2 · tenure tend = d["Last Day"].fillna(SNAP) d["t"] = ((end - d["Date Join"]).dt.days / 365.25).clip(lower=0) print(d["t"].describe()[["min", "50%", "max"]].round(2).to_dict()) # {'min': 0.07, '50%': 4.2, 'max': 14.37} -> the curve is defined out to about 14 years

The event flag. One if the stay was seen to end, zero if it is still running at the snapshot. This column is what separates survival analysis from a tenure average.

★ Step 3 · event flagd["event"] = d["Last Day"].notna().astype(int) print(d["event"].value_counts().to_dict()) # {0: 861, 1: 210} exits = d.loc[d["event"] == 1, "t"] print(round(exits.median(), 2), round((exits < 1).mean(), 3), round((exits < 2).mean(), 3)) # 1.52 0.295 0.61 -> tenure at exit: median 1.52 y, 29.5% under one year, 61.0% under two

Sort, and let the at-risk count fall. Every row, exit or censored, removes one person from the at-risk count after it is processed. Only exits change S.

★ Step 4 · sort and walkkm = d[["t", "event"]].sort_values("t").reset_index(drop=True) at_risk = len(km) # 1071 at t = 0 S = 1.0 surv = [] for t, e in zip(km["t"], km["event"]): if e == 1: S *= 1 - 1 / at_risk # one step down, sized by who was still there at_risk -= 1 # exit or censored, one fewer at risk surv.append(S) km["S"] = surv

Read the curve at four tenures. S at t is the last value at or before t. These are the four numbers in Part 2, and they are correct for the table and wrong for Apex.

★ Step 5 · read S(t)for h in (1, 2, 3, 5): seen = km.loc[km["t"] <= h] print(f"S({h}) = {seen['S'].iloc[-1]:.3f} exits seen so far {int(seen['event'].sum()):3d} at risk after {len(km) - len(seen):4d}") # S(1) = 0.940 exits seen so far 62 at risk after 946 # S(2) = 0.872 exits seen so far 128 at risk after 803 # S(3) = 0.813 exits seen so far 180 at risk after 670 # S(5) = 0.778 exits seen so far 206 at risk after 441

Show the shape that causes the bias. Every exit's date falls inside one six-month window, while the at-risk years stretch back to 2012. Print both facts; they are the proof.

★ Step 6 · the shape of the inputprint(d["Last Day"].min().date(), d["Last Day"].max().date()) # 2026-01-01 2026-06-17 print(d["Date Join"].min().date()) # 2012-02-16 print("person-years at risk in the denominator:", round(d["t"].sum())) # 5059 print("months in which an exit could be seen:", 6) # every year of exposure before 2026 is in the denominator and could not reach the numerator

Print the honest sentence, and save the curve with its warning in the filename. The sentence has three parts and the code writes all three, so that nobody downstream sees the number without them.

★ Step 7 · the sentences1 = km.loc[km["t"] <= 1, "S"].iloc[-1] print(f"On the people present in 2026, one-year survival is {s1:.3f}, " f"biased UPWARD because no exit before 2026-01-01 is in the table; " f"the size of the bias cannot be measured from this table.") km.to_csv("km_survivor-table_snapshot-2026-07-01.csv", index=False)

Keep the word "survivor-table" in the filename. A curve saved as km.csv will be read next quarter as the truth. A curve saved with the snapshot date and the words "survivor table" carries its own warning into every spreadsheet that opens it. When the event log exists, the same seven steps produce km_event-log_from-2012-02-16.csv, and the two files can sit side by side, which is the first time the size of the bias becomes a number.

Homework

Before deep-dive session 3

30 min
  1. Run the seven steps on Apex, then on your own export. Before running it on yours, write down the snapshot date and the earliest exit the export can see; the gap between the earliest join and the earliest exit is the span of exposure that cannot reach the numerator.
  2. Move the snapshot date to 1 April 2026 and re-run. Exits after 1 April become censored stays. Write down which of the four readings moved and in which direction, and why the at-risk counts did not change at t = 0.
  3. Split the curve by entity, four staircases from the same loop. Write one sentence on why comparing them is still a comparison of four biased curves, and what would have to be true for the ranking to survive the bias.
  4. Specify the event log. Four columns, a start date, and the two routes to get it. Bring the specification; deep-dive session 3 turns the same 210 exits into reasons and puts them beside a national benchmark.
Check yourself

Three questions

Clear all three and deep-dive session 2 gets stamped in your passport.

1 · A person joined in 2021 and is still employed at the snapshot on 1 July 2026. In the Kaplan-Meier loop, what does their row do?

Censoring is a lower bound, not a gap. The row is at risk up to its own t and is then removed from the count without an event, which is why censoring alone can never lower a survival curve.

2 · The curve on Apex reads 0.940 at one year. Why is that number too high, and by how much?

The estimator is fine; the input is a survivor table. Exposure that cannot produce a visible event drives the fraction down at every t. The size depends on rows that do not exist.

3 · Tenure at exit has a median of 1.52 years, and the curve says 87.2 percent of stays survive two years. What does the pair tell you?

Both numbers are correct for the rows present. The tension between them is a signature of the missing rows, and it is the nearest thing to a size estimate the table can offer: not a number, but a direction with weight behind it.

Sources covered

What this session teaches, and where it came from

✓Kaplan and Meier 1958, "Nonparametric Estimation from Incomplete Observations", Journal of the American Statistical Association 53(282) 457-481Primary; the product-limit estimator, written as a loop in the build-along
✓Tenure as time-to-event; censoring as a lower bound; the at-risk countStandard practice, no single source
✓Apex survival 0.940 / 0.872 / 0.813 / 0.778 at 1 / 2 / 3 / 5 years; tenure at exit median 1.52, 29.5 percent under one year, 61.0 under two; active tenure median 5.01Computed from the constructed Apex table that ships with these pages; the curve is biased upward and the page says so
✓The survivor-table bias: direction proved from the shape of the export, size not measurableStandard practice; the argument uses only rows present in the table and the span of dates they carry
◐Cox 1972, "Regression Models and Life-Tables", Journal of the Royal Statistical Society Series B 34(2) 187-202Named in self-study as the regression on the hazard; not fitted, because a survivor table would bias it
◐The customer version of survival, and the event log that fixes the inputLinked to the Customer Retention course; the log is specified, not built

Every fact on this page, and its verification tier, is recorded in the course's source map.

Deep-dive session 2 cheat sheet · pin this

Two columns per stayt = years from join to exit or to the snapshot; event = 1 if the exit was seen, 0 if censored.
Censored is a lower boundAt risk up to its own t, removed without an event. Never moves S down.
The stepAt each exit, S *= 1 - 1/at_risk. Then at_risk -= 1 for every row.
Apex reads0.940 / 0.872 / 0.813 / 0.778 at 1 / 2 / 3 / 5 years. All too high.
The proofDenominator holds years before 2026; numerator can only hold H1 2026 exits. Hazard low, S high.
The honest sentencePopulation, direction of bias, size unknown. Then name the event log that would fix it.