learn-customer-retention-with-phoebe / Analyst session 7 of 8
Learn Customer Retention with Phoebe · Analyst track · Session 7 of 8

Churn prediction that helps

A churn model is the easiest impressive artifact in data science and the easiest to get silently wrong. Today you build one that survives contact with reality: a training frame whose feature window strictly precedes its label window, an evaluation that does not let accuracy lie about imbalanced classes, SHAP explanations a stakeholder can act on - and the honesty core most write-ups skip: the customers your model ranks riskiest are often exactly the ones an offer cannot save.

🟠 Analyst track Data analysts · Python Build-along: XGBoost + SHAP 45 min
0-3 · Welcome 3-20 · Framing without leaks 20-40 · Model, explain, price 40-45 · Q&A
Part 0

Where this session sits

Seventh of eight analyst sessions, one running artifact: the retention report a8 ships. You already know what retention means (a1), you built the cohort triangle (a2) and its viz gallery (a3), you segmented who retains (a4), modeled lifetimes (a5), and scored P(alive) per customer (a6). Today adds the last modeling piece: a supervised churn model that names WHO is at risk and WHY - risk scores the capstone folds into its recommendations. Work along in the companion notebook notebooks/a7_churn.ipynb on the repo's synthetic data, where the churn generator is known, so a leak shows up as an impossible score.

Live - presented in session Self-study - read after class ★ Build-along - run the code yourself Sources covered
★ What you walk out with today A leak-proof windowed training frame (features strictly before a cutoff, labels strictly after), a logistic baseline plus an XGBoost workhorse evaluated on precision-recall instead of the accuracy lie, SHAP plots that translate the model into stakeholder language, and the Ascarza discipline: spend the retention budget on persuadables, not on the doomed or the already-loyal.
Part 1 · covers label/feature windows + the leakage traps

The framing IS the model 10 min live

Before any algorithm, you make one decision that determines whether the model is real: pick a cutoff time t. Features may use behavior up to and including t. The label is whether the customer churned in a window strictly after t - within 30, 60, or 90 days, matching a1's forward-looking law. Any feature that peeks past the cutoff is leakage: the model aces validation because it is reading the answer key, then dies in production where the future genuinely is not available yet.

The two windows: what the model may see vs what it must predict past future Feature window Behavior up to t: orders, tenure, tickets, payment failures, days-since-last measured AT t cutoff t Label window Churned within 90 days after t? Only events in (t, t + 90d] define the label. Nothing else. exit_survey_score: post-churn data Any feature that crosses the cutoff aces validation and dies in production - the leak IS the lie.
🔍 Click to zoom - features left of the cutoff, labels right of it, and nothing may cross the line
LiveSpot the leak: 8 features, 3 of them lie6 min▶

A junior analyst proposes this feature list for a 90-day ecommerce churn model with cutoff t. Three features leak. Find them before reading on - in session we vote before the reveal.

  1. tenure_days - days from first purchase to t
  2. orders_last_90d - orders in the 90 days before t
  3. days_since_last_order - computed when the training script runs (today)
  4. support_tickets_180d - tickets opened in the 180 days before t
  5. exit_survey_score - the churn survey answer, NaN for active customers
  6. payment_failures_90d - failed charges in the 90 days before t
  7. sessions_week_after_cutoff - site visits in the 7 days after t
  8. avg_order_value - mean order value over orders before t
The reveal: 3, 5 and 7 #3 is the classic silent leak: days-since-last computed at training time instead of at the cutoff encodes everything that happened AFTER t - churners trivially have huge values. Compute it as (t - last order before t). #5 is a field only populated at churn: exit-survey answers, final-invoice flags, cancellation reasons - the label wearing a costume. If a column is NaN for exactly the customers who did not churn, it is the answer key. #7 is post-cutoff activity, the label window itself. The other five are clean because every one of them can be computed on the day t falls, using only the past.
★ Build the windowed training frame - the cutoff does all the workimport pandas as pd events = pd.read_csv("data/ecommerce_orders.csv", parse_dates=["order_date"]) cutoff = pd.Timestamp("2011-06-01") past = events[events["order_date"] <= cutoff] # feature window future = events[(events["order_date"] > cutoff) & (events["order_date"] <= cutoff + pd.Timedelta(days=90))] feats = past.groupby("customer_id").agg( tenure_days = ("order_date", lambda s: (cutoff - s.min()).days), orders_90d = ("order_date", lambda s: (s > cutoff - pd.Timedelta(days=90)).sum()), days_since_last = ("order_date", lambda s: (cutoff - s.max()).days), avg_order_value = ("amount", "mean"), ) feats["churned_90d"] = (~feats.index.isin(future["customer_id"])).astype(int)

One frame per cutoff. Every aggregate above reads only past; the label reads only future. That physical separation - two dataframes, not one - is the cheapest leakage insurance there is.

The eight proposed features against cutoff t: three cross the line before t cutoff t after t 1 · tenure_days clean 2 · orders_last_90d clean 3 · days_since_last (today) LEAK computed today, not at t - encodes the future 4 · support_tickets_180d clean 5 · exit_survey_score LEAK only exists after churn 6 · payment_failures_90d clean 7 · sessions_wk_after_cutoff LEAK the label window itself 8 · avg_order_value clean The test for every feature: could the warehouse compute this value on the morning of t? Needs today = leak.
🔍 Click to zoom - the eight features as bars against the cutoff: the three struck in red read the future
Part 2 · covers baseline-then-boost, imbalance, SHAP

Baseline, boost, explain ★ 12 min · everyone builds

Always fit logistic regression first: its coefficients are a sanity check (does more tenure really lower risk? do payment failures raise it?) and a floor the fancy model must beat. Then XGBoost or LightGBM as the workhorse. And because churn is imbalanced, accuracy is a lie: on 5% churn, a model that predicts "nobody churns" is 95% accurate and catches zero churners. Precision-recall is the honest lens - precision is your wasted intervention budget, recall is your missed churners.

Live★ Build-along: split by time, fit, evaluate honestly8 min▶
★ Split BY TIME, never randomly - random rows leak the futuretrain_cutoffs = [pd.Timestamp("2010-12-01"), pd.Timestamp("2011-03-01")] test_cutoff = pd.Timestamp("2011-06-01") # strictly later than training train = pd.concat([build_frame(events, c) for c in train_cutoffs]) test = build_frame(events, test_cutoff) X_tr, y_tr = train.drop(columns="churned_90d"), train["churned_90d"] X_te, y_te = test.drop(columns="churned_90d"), test["churned_90d"] # NEVER train_test_split(shuffle=True) here: rows from the same calendar # period land on both sides, and validation quietly rehearses the future.
★ Logistic baseline, then the XGBoost workhorsefrom sklearn.linear_model import LogisticRegression from xgboost import XGBClassifier base = LogisticRegression(max_iter=1000).fit(X_tr, y_tr) print(dict(zip(X_tr.columns, base.coef_[0].round(3)))) # signs make sense? model = XGBClassifier( n_estimators=300, max_depth=4, learning_rate=0.05, scale_pos_weight=(len(y_tr) - y_tr.sum()) / y_tr.sum(), # imbalance eval_metric="aucpr", ).fit(X_tr, y_tr)
★ Evaluate on precision-recall - accuracy stays in the drawerfrom sklearn.metrics import classification_report, average_precision_score proba = model.predict_proba(X_te)[:, 1] print(classification_report(y_te, proba >= 0.5)) print("PR-AUC:", round(average_precision_score(y_te, proba), 3)) # Compare against the do-nothing baseline: PR-AUC of a random ranker # equals the churn rate itself. Beat THAT, not 0.5.
★ SHAP: global drivers, then one customer's storyimport shap explainer = shap.TreeExplainer(model) shap_values = explainer(X_te) shap.summary_plot(shap_values, X_te) # beeswarm: which features drive risk shap.plots.waterfall(shap_values[42]) # why THIS customer scores 0.83

The beeswarm answers the stakeholder question ("what drives churn here?") once, globally. The waterfall answers the CRM question ("why is this customer flagged?") per row - that per-customer explanation is what makes a score actionable instead of spooky.

Precision vs recall as the threshold moves: two costs, one slider threshold 0 (contact everyone) threshold 1 (contact no one) Precision rises Recall falls your threshold lives here Too low: wasted offers (precision cost) Too high: missed churners (recall cost)
🔍 Click to zoom - the threshold is a budget dial: precision prices wasted offers, recall prices missed churners
Self-studyFeature ideas that survive production4 min read▶

The features that keep working are the boring ones any pipeline can recompute at any historical cutoff:

  • Tenure at the cutoff - months since first purchase or install.
  • Usage trend deltas - this week's activity minus last week's, week-over-week, computed inside the feature window. A falling delta is often the earliest honest signal.
  • Support tickets - count and recency before the cutoff; pair with resolution time if you have it.
  • Payment failures - failed charges before the cutoff; among the strongest involuntary-churn predictors.
  • Days-since-last, computed at the cutoff - the same recency that powers a6's BG/NBD, and the #1 leak when computed lazily at training time.

Rule of thumb: for every feature, ask "could the data warehouse have produced this exact value on the morning of the cutoff date?" If the answer needs the word "today", it leaks.

Part 3 · covers the Ascarza caveat + intervention economics

Prediction is not prevention 8 min live

Here is the finding that should be stapled to every churn model, from Eva Ascarza's research: the customers most likely to churn are often the least persuadable. Ranking by risk and mailing the top decile wastes budget twice - on "lost causes" no offer can save, and on "sure things" who would have stayed anyway. What moves money is targeting persuadables: customers whose behavior actually changes because you contacted them. That is uplift modeling - two models (treated vs control) or one model with a treatment flag - and it needs experiment data: intervention design is experiment territory - see learn-experimentation-with-phoebe.

LiveThe four customer types - and the one worth your budget5 min▶

Cross two counterfactuals - what they do if left alone, what they do if contacted - and every customer lands in one of four boxes:

  • Sure things - stay either way. Every discount sent here is margin set on fire.
  • Persuadables - churn if ignored, stay if contacted. The ONLY box where the offer causes retention. All budget goes here.
  • Lost causes - churn either way. Usually the top of your risk ranking, which is exactly the trap.
  • Sleeping dogs - stay if ignored, churn BECAUSE you contacted them. A renewal reminder that wakes someone who forgot they were subscribed. Do not disturb.

And the threshold question changes shape: not "which cutoff maximizes F1" but which cutoff maximizes (margin saved per rescued customer) minus (cost of offer x customers contacted). A $5 winback email tolerates low precision; a $50 discount does not. Price the threshold from the intervention economics, then read the precision-recall curve at that price.

The persuadability grid: budget flows to exactly one box left alone: STAYS left alone: CHURNS contacted: STAYS contacted: CHURNS Sure things Stay either way. An offer here is pure cost: discounting the loyal. Persuadables Churn if ignored, stay if contacted. The only box the offer actually changes. Sleeping dogs Churn BECAUSE you called. Contact reminds them to cancel. Do not disturb. Lost causes Churn either way. Highest risk score, zero return - the risk-ranking trap. Retention budget Offers, winbacks, saves - spent on uplift, not risk. Risk models sort top-to-bottom by churn probability; uplift models sort left-to-right. Money lives left-to-right.
🔍 Click to zoom - sure things, persuadables, lost causes, sleeping dogs: budget arrows point at one box only
Real world

Winback economics, priced honestly. Reactivation emails realistically win back 10-30% of lapsed customers - a real but bounded lever. The combination that works in practice: a7's risk score says who is fading, a6's P(alive) says who is still plausibly alive, and a4's value segments say who is worth an offer. Send the winback to high-value, fading-but-alive customers - not to the whole top of the risk ranking, where the lost causes live and the budget goes to die.

Homework

Try it yourself - this week ◐ 40 min total

Sources covered

What this session draws on

ML churn modeling is standard supervised learning; what makes it churn-specific is the windowing discipline and the intervention caveat. The working core:

✓Leakage and windowing discipline - feature window strictly precedes label windowPart 1 · the spot-the-leak drill
✓The XGBoost + SHAP workflow (logistic baseline, time-based split, PR evaluation)Part 2 · build-along
✓Eva Ascarza - retention futility research: prediction is not prevention, target persuadablesPart 3 · the four-types grid
✓Intervention-economics thresholding (offer cost vs margin saved, not F1)Parts 2-3 · the budget dial
◐Uplift modeling implementation (two-model, treatment-flag)Caveat only; full method: learn-experimentation-with-phoebe
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · A colleague splits the churn training frame with train_test_split(shuffle=True) and reports a stellar validation score. Why is the split itself a leak?

Churn frames are time-stamped: customers observed under the same market conditions, promotions and seasonality end up in both train and test, so the model is graded on a period it effectively trained on. Split by time - train on older cutoffs, test on a strictly later one - so validation simulates deployment.

2 · Your churn rate is 5% and a model reports 95% accuracy. What have you learned about the model?

On imbalanced classes, accuracy mostly measures the class ratio. Precision-recall is the honest lens: precision prices wasted intervention budget, recall prices missed churners, and PR-AUC should be compared against the churn rate itself (a random ranker's PR-AUC), not against 0.5.

3 · Under uplift thinking, who gets the retention offer?

Ascarza's finding: the riskiest customers are often lost causes no offer can save, and blanket saves also pay sure things who would stay anyway - while sleeping dogs churn BECAUSE you contacted them. Only persuadables convert offer spend into retained revenue, and only experiment data can identify them.

Analyst session 7 cheat sheet · pin this

The framing lawFeatures from behavior up to cutoff t; label from (t, t+30/60/90d]. Any overlap is leakage. The framing IS the model.
The classic leaksThe cancellation event itself · post-cutoff activity · churn-only fields (exit surveys, final-invoice flags) · days-since-last computed at training time.
Split by timeTrain on older cutoffs, test on a strictly later one. Random splits let validation rehearse the future - the score is a lie.
Accuracy lies95% accuracy on 5% churn can catch zero churners. Precision = wasted offers; recall = missed churners; benchmark PR-AUC against the churn rate.
Baseline then boostLogistic regression first - coefficient signs are the sanity check and the floor. Then XGBoost/LightGBM with scale_pos_weight for imbalance.
SHAP two wayssummary_plot beeswarm = global drivers for the stakeholder deck; waterfall = why THIS customer scored 0.83, for the CRM.
Prediction is not preventionSure things / persuadables / lost causes / sleeping dogs. Budget goes to persuadables only - uplift, not risk ranking (Ascarza).
Price the thresholdChoose the cutoff where margin saved beats offer cost x contacts - intervention economics, never F1. Cheap emails tolerate low precision; discounts do not.