Qini curves and who to treat

The S-, T- and X-learners of session 7 each return the same object: an estimated individual effect, one row per customer, in the units of the outcome. For Kestrelmark that is extra spend caused by receiving the email. Across all 5,000 customers the true average effect is 6.23, and the distribution around that average is the whole point, because 6.23 is an average of numbers that are mostly not 6.23.

Leader session 5 splits the same population four ways, and in this data the split has a name in the columns: recency. Sure things, still active, are 22.4 percent of the base and the email is worth +2.10 to them. Persuadables, lapsed 30 to 90 days, are 20.4 percent and worth +29.09. Lost causes, dormant a year, are 49.3 percent and worth +1.04. Sleeping dogs, customers who cancelled once already, are 7.9 percent, and the email costs them 8.87 units each.

Session 7 measured how far that goes: recency alone explains the individual effect with an R-squared of 0.986. Which is to say the group is recoverable and the person is not, because inside every group there is still a 3.0 coin flip that nobody can predict. Tonight only needs the group, which is fortunate, because the group is all any model can give you.

• 3,500 customers to fit, 1,500 to score. The meta-learner from session 7 never sees the 1,500. Every number tonight comes from the held-out block.
• Both arms live in the held-out block. The curve compares treated against untreated customers inside each slice of the ranking, so a slice with no controls in it produces nothing.
• The ranking is predicted, the response is actual. Order the held-out customers by predicted uplift; measure the gain from what those customers actually did. A curve built from predictions on both axes is a picture of the model agreeing with itself.
• The truth is available here and nowhere else. Kestrelmark comes from a written structural model, so tonight's curve can be checked against a known answer. Your own curve cannot be, which is the reason the held-out discipline is not optional.

A likelihood-to-buy model is fit on outcomes. It learns that customers with high tenure, recent activity and a full basket tend to spend, and it ranks them first. Nothing in that fitting procedure ever sees a comparison between a treated customer and an untreated one, so nothing in the resulting score is about the email at all. It is a good model of who spends and it is silent on the question being asked.

An uplift model is fit on the difference between arms. Whatever the meta-learner does internally, the quantity it targets is the change in spend caused by the email for a customer with those characteristics. The two scores are related only through whatever accidental overlap exists between "spends a lot" and "responds to a nudge", and at Kestrelmark that overlap is weak enough to be worth 0.94 against 0.53 on the scale Part 2 builds.

Send down a likelihood-to-buy list and the money goes to sure things. These are customers who were going to purchase whether or not the email arrived, so the discount, the send cost and the attention are spent to produce an outcome that was already coming. The campaign report will look excellent, because the emailed group really did spend a lot, and every unit of that spend belongs to the customer rather than to the campaign. Session 1 priced exactly this confusion at Kestrelmark: 10.54 reported against 6.23 real.

• Sure things get the budget. 22.4 percent of the population, high baseline spend, an effect of +2.10. They are at the top of the likelihood list by construction.
• Persuadables carry the business case. 20.4 percent of the population at +29.09 each, an order of magnitude above every other group. Many of them are low-baseline customers precisely because they are drifting, which is what puts them low on a likelihood ranking.
• Lost causes waste less than they look. Half the population, 49.3 percent, at +1.04. They are cheap mistakes: the send cost, and almost nothing else.

The arithmetic is unforgiving, and Part 2 measures it. At the ninth decile the uplift ranking has banked 10,698 units of incremental spend on the held-out block. At the same send volume the likelihood ranking has 9,644, and it has nowhere left to go but the blanket total.

Sleeping dogs are the customers a campaign actively drives away. 7.9 percent of Kestrelmark's base, at an effect of -8.87 each: the email reminds a semi-dormant subscriber that the subscription exists, and they cancel, downgrade, or simply stop drifting pleasantly along. Their existence is not exotic. Any contact that carries a cancel link, a price, or a reminder of a commitment has a group like this.

The dangerous part is that sleeping dogs look engaged. They have history, they have spend, they open things. A likelihood-to-buy model reads that history, scores them high, and puts them near the top of the send list, which is the worst place they could be. On the held-out block, 111 of the 1,500 customers are genuinely harmed by the email, and no report built on outcomes alone will ever surface them.

The two lists are not always almost disjoint, and knowing when they converge saves the argument. They agree when the treatment effect is roughly proportional to baseline outcome, which happens with percentage-style interventions: a ten percent discount on a customer who buys more is worth more in absolute units. In that world, ranking by predicted spend is a decent proxy for ranking by uplift and the Qini curves of the two rankings sit close together.

They diverge when the effect is roughly flat in absolute units, when the treatment works by removing a barrier that high-baseline customers never had, or when a harmed subgroup is concentrated among the engaged. Kestrelmark is the second kind: the email is a reminder, reminders do nothing for people already active, and the reminder itself irritates a slice of the customers who have cancelled before. Check which world you are in before assuming either result.

Sort the held-out customers by predicted uplift, highest first. Walk down the list in deciles. At each stopping point, take everyone above it and ask what the treated ones actually earned compared with the untreated ones in the same slice, scaling the control group up to the size of the treated group so the two are comparable. That is the cumulative incremental gain at that share, and it is measured from what people did, not from what the model predicted they would do.

• The x axis is share of the population, not a threshold. Reading it as a budget is the point: at the fourth decile you are sending to four rows in every ten of the held-out block.
• The y axis is cumulative and in outcome units. Units of spend here, so the height is directly comparable to the cost of sending.
• The right-hand endpoint is fixed. Target everybody and the ranking is irrelevant, so every curve on the same data must end at the same value. Kestrelmark's is 9,729 units, and both rankings arrive there, because treating everybody is the same campaign whatever order you did it in. The endpoint is a property of the campaign, not of the model.
• The diagonal is random targeting. A random s percent of the population captures s percent of the total gain, in expectation, which is a straight line from the origin to that fixed endpoint.

The area between the curve and the diagonal is the Qini coefficient. Larger is better, zero means the ranking is no better than shuffling the list, and negative means the ranking is actively inverted. On Kestrelmark's held-out block the uplift ranking scores 4,061 and the likelihood-to-buy ranking scores 2,290. Divided by the 4,298 a perfect ranking would achieve on the same data, those become 0.94 and 0.53, numbers between 0 and 1 that can be compared across models.

A curve that peaks and then falls is saying something specific and easy to miss. Gain accumulates while the customers being added have positive actual response. It flattens where the added customers do nothing. It falls where the added customers are harmed. So the peak is not a modelling artifact to be smoothed away: it is the recommendation. Kestrelmark's uplift curve peaks at the ninth decile, 10,698 units, and treating the last decile as well gives about 1,000 of them back, arriving at the 9,729 a blanket send would have produced. On a finer grid than deciles the true optimum is the top 92.5 percent at 10,731, which is 1,002 units above the blanket send.

Everything above is gross incremental gain. Leader session 5 prices a Kestrelmark send at 7.00 fully loaded, and once that bar is on the page the four groups sort themselves without a curve at all.

• Persuadables clear it, and only persuadables. +29.09 against 7.00.
• Everybody else loses money on every send. Sure things return 2.10, lost causes 1.04, sleeping dogs -8.87. All three sit below the cost line in the figure above.
• The blanket campaign is net negative. Run at that cost across the base it is -3,848. Sending only to the 20.4 percent persuadable segment is +22,578.

That is the same lesson as the peak, stated in money rather than in units of spend. The average effect of 6.23 is positive, the gross gain of 9,729 is positive, and the programme as it is actually run destroys value. Nothing in a campaign report built on the average can see it.

The honest limits are worth more than the curve, because they are what stops it being quoted as proof.

• It is built from estimated uplift. Every identification assumption from sessions 2 and 3 travels straight into it. If a back-door path was left open, the per-customer effects are biased and so is the ordering built from them. A beautiful Qini curve computed on a mis-specified adjustment set is a confident picture of the wrong ranking.
• Fit and score on the same rows and it flatters itself. The model that produced the ranking has already seen those outcomes, so the top slices look better than they are. Held-out, always, and a curve reported without a stated split should be assumed to be in-sample.
• Both arms must be present in every slice. Squeeze the deciles too fine, or rank in a way that correlates with who was treated, and some slices lose their control group. The gain in those slices is noise wearing a number.
• The peak is estimated too. It moves between resamples, and on 1,500 rows it can move by a decile. The decile grid puts it at 90 percent and the finer grid at 92.5 percent, which is a useful reminder that it is a region rather than a point. Say so when you hand over the share.
• Correcting a contaminated comparison is a different job. Reweighting or modelling a treated-against-control comparison to remove the remaining imbalance belongs to learn-experimentation-with-phoebe. What the graph decides is whether the comparison is correctable at all.

The similarity is real and it is what makes the plot legible on sight: cumulative capture against cumulative population, a diagonal for chance, an area for a summary. An analyst who reads ROC curves needs no training to read a Qini curve, which is most of the reason the format won.

• An ROC curve cannot go down. Every extra case adds a non-negative amount of capture. A Qini curve can, and the descent carries the most actionable information on the page.
• The y axis is in outcome units, not a rate. That is a feature: it converts directly against a send cost, so a cost line drawn on the same axes gives a net-value optimum rather than a gross one.
• There is no ground truth per row. An ROC curve is scored against labels that exist. No customer's individual effect is ever observed, so the curve is scored against group differences and inherits their noise.