CATE and uplift

A budget that covers the whole list makes heterogeneity a curiosity. A budget that covers a fifth of it makes heterogeneity the entire problem, because the question is no longer whether the campaign works but which fifth to spend it on. Those two questions can have opposite answers: a campaign with a healthy average can be worth running on half the list and worth withholding from the other half, and the average will never say so.

Kestrelmark is that case, measurably. The file carries a recency segment, and the email is worth a different amount in each of its four values: +2.10 to a customer who is still active, +29.09 to one who lapsed 30 to 90 days ago, +1.04 to one dormant a year, and -8.87 to one who has already cancelled once. Those four average to the 6.23 the bench recovers, and the 6.23 describes none of them.

There is a second layer underneath, and it is the one that sets the ceiling on every model tonight. Inside each of those four groups the effect still takes one of two values, 3.0 apart, according to whether the email is what tipped that person into opening. Nothing observed predicts which. So the honest shape of the answer is: the group is knowable, the person is not.

• ATE - the average treatment effect over the whole population. 6.23 here.
• CATE - the average effect among customers sharing a given set of covariate values. A function of those covariates. At Kestrelmark, conditioning on recency alone gives +2.10, +29.09, +1.04 and -8.87.
• Individual effect - one person's own difference between two worlds. Session 6, and it needs the structural equations.
• Effect modifier - a variable that changes the size of the effect. It need not cause the treatment, and if it does not, it is not a confounder. recency is the one here.
• Uplift - the industry word for a predicted CATE used for targeting. Same object, different room.
• Meta-learner - a recipe that turns any ordinary regression algorithm into a CATE estimator by arranging what it is fitted on.
• Base learner - the ordinary algorithm doing the fitting inside a meta-learner. Gradient boosting, ridge, a forest: the choice matters and it is a separate literature.

The distinction that causes the most trouble in practice is the second against the third. A CATE describes a group defined by covariates and can be estimated from data. An individual effect describes a person and cannot. A targeting system that quietly slides from one to the other is making rung-3 claims on rung-2 evidence.

Write the estimand out and the change is obvious: instead of one difference of means over the whole list, there is a difference of means inside every covariate cell, and the collection of those differences is the object. Averaging the function over the population returns the 6.23 from the bench, which is the first and cheapest check any CATE model should pass.

• The covariates that define the cells are a choice. Condition on season alone and you get two numbers. Condition on recency alone and you get the four that matter here. The finer the conditioning, the closer to an individual and the noisier every cell.
• The rule is not "only what you adjusted for". It is "nothing that was forbidden". Colliders, mediators and anything downstream of the treatment stay out, for exactly the reasons session 2 gave. A pre-treatment variable that does not cause the treatment was never forbidden, and here it turns out to be the whole story.
• The function can be flat, and it is worth knowing that in advance. A finding that the effect does not vary with anything you measured is a result, not a failed project. On this data it is emphatically not flat: the widest gap between two segments is 29.09 against -8.87.

This is the point that decides what goes into the feature list, and it is the one most often got wrong in both directions. recency shifts a customer's baseline spend and it sets how much the email is worth to her. What it does not do is influence whether she was sent the email. No arrow runs from recency into the treatment, so it opens no back door, and the adjustment set defended in session 3 is unchanged: {season, tenure}.

That is the difference between the two jobs a covariate can hold. Closing a back door is about identification and the set is forced on you. Describing how the effect varies is about the estimand you chose, and the set there is a modelling decision. A variable can hold the second job without the first.

This is the property that makes the whole area strange, and it is worth saying slowly. In ordinary supervised learning every training row carries the answer: the price the house sold for, the class of the image. A model is fitted to that answer and scored against held-out copies of it.

There is no such column here. A row records what the customer spent under the treatment she received, and the treatment effect is a difference between that and what she would have spent under the other one. No row contains both. The quantity being modelled has never been observed for a single customer in the table, and it never will be.

• So there is no loss function on the target. Every method builds an indirect construction: a difference of two fitted surfaces, a transformed outcome, a pseudo-effect. What gets minimised is always a proxy.
• And there is no honest held-out score. A validation set has the same missing column, so the usual habit of trusting the leaderboard does not transfer. Session 8 covers what can be done instead by ranking rather than by scoring.
• Which is why model comparison here is unusually hard, and why running three learners and comparing their disagreement is not a luxury. Disagreement is one of the few signals available.

Because Kestrelmark is generated from a written model, the per-person effect can be computed rather than estimated, which is what session 6 did. Doing it for all 5,000 customers gives eight values and no others. Every one of them is a segment's own direct effect, plus 3.0 when the email is the thing that tipped that person into opening.

Regress that column on recency alone and the R-squared is 0.986. Almost all of the variation in who the email helps is one field in the warehouse, which is unusually kind and is the reason a CATE model has anything to find here at all. What is left over is the part no feature can reach.

• The segment effect is the ceiling. A perfect model, given every observed column, converges on +2.10, +29.09, +1.04 and -8.87. That is the best any of tonight's three learners could possibly do, and it is enough for every decision session 8 makes.
• The individual effect is out of reach, and not for want of data. Two persuadables with identical covariates get identical predictions, while one of them truly gets 28.0 and the other 31.0. What separates them is whether the email happened to be the thing that made them open it, and that is an unobserved draw rather than a missing column.
• So the right unit of a targeting promise is a group. "Customers who lapsed 30 to 90 days ago respond at 29.09" is defensible. "This customer will respond at 29.09" is not, and the second sentence is what a ranked file with a name against every score invites people to say.

Estimating the effect per person is tonight. Turning it into a send list is session 8. It is still worth seeing why the estimate matters, because the gap is not marginal.

Leader session 5 puts Kestrelmark's fully loaded cost at 7.00 per send, in the same units as spend. Against that bar, only the persuadable segment clears: 29.09 against 7.00. Sure things at 2.10, lost causes at 1.04 and sleeping dogs at -8.87 all lose money, the last group twice over. A blanket send to the whole file is worth -3,848; sending only to the 20.4 percent persuadable segment is worth +22,578.

That comparison is the payoff of estimating a function instead of a number, and it is the reason to get the segments right rather than merely to get a ranking. How to choose the depth of the send, how to score one ranking against another, and how to read the curve that finds the stopping point, all belong to session 8, which consumes the per-customer predictions this session produces.

Fit one model on the covariates with the treatment included as an extra feature, then score every row twice, once with that feature set to 1 and once with it set to 0. The difference is the predicted effect. It is the simplest recipe, it uses all the rows for one fit, and it is usually the first thing anybody writes.

• The documented weakness is regularisation. Because the base learner is normally regularised, the estimated treatment effect is biased toward zero, and the treatment column is just one feature competing with all the others for a share of the penalty budget.
• If the treatment is weak relative to the other features, it can be ignored entirely. A tree-based learner may never split on it; a penalised linear one may shrink its coefficient to nothing. The output is then a predicted effect of zero everywhere, which looks like a clean finding of no heterogeneity.
• It is worst exactly where it matters. A strong treatment survives the penalty; a marginal one, which is the case where targeting could actually help, is the one that disappears.

The failure is silent in the same way a mediator is silent in session 2. Nothing errors, the outcome fit looks respectable, and the targeting recommendation is "it makes no difference who you send to".

Fit one model on the treated rows only and another on the control rows only, then difference their predictions for every customer. The treatment can no longer be shrunk away, because it is never a feature: it is the reason there are two models.

• It avoids missing a weak effect, which is the specific fault it exists to fix, and that is enough reason to always run it alongside an S-learner.
• It still suffers regularisation, now in both fits at once, and the two penalties do not cancel. Their difference can wander in ways neither surface would on its own.
• And it fits two models on smaller samples. Each learner sees only its own arm, so where one arm is thin the surface there is poorly estimated and the difference inherits that noise directly.

Three moves. First, impute the counterfactual outcome for every row using the other arm's model: for a treated customer, what the control surface says she would have spent. Second, take the difference between her observed outcome and that imputed one, which gives a pseudo-effect for every row, and regress on those. Third, combine the two resulting fits, weighting them by propensity so that each region leans on the arm that has more data there.

• It is particularly suited to treated and control imbalance, which is the norm outside experiments. Observational campaign data is almost never balanced, and this is the recipe that was designed around that fact.
• The propensity model here is a component, not a topic. It appears as a weight between two fits. Fitting one as an estimator in its own right, with inverse probability weighting or doubly robust estimation on top, belongs to learn-experimentation-with-phoebe and is deliberately not done here.
• The cost is two extra moving parts. The imputation is a model, the weighting is a model, and an error in either propagates into every per-person number. More structure buys more places to be wrong.

Fitting any of these three well, choosing base learners, tuning them without a validation target, and getting honest variance out of them, is a modelling exercise with its own substantial literature; what matters here is knowing which recipe fails in which direction.

Nothing in any of these recipes revisits identification. They take an adjustment set as given and produce a function on top of it, which means the covariate set on the inside is the same object that sessions 2 and 3 spent their time defending. Put the wrong columns in and the model does not fail loudly; it personalises the failure.

• A collider in the feature set is still a collider. Adding the loyalty score moves the average from 6.56 to 5.26, and there is no reason it improves per segment. It biases every cell and the biases need not point the same way.
• A mediator changes the estimand at every level of granularity. Including the open flag drops the average to 5.48, close to the mean direct effect of 5.16, so every predicted number becomes a conditional direct effect and the whole surface answers the subject-line question rather than the campaign question.
• A confounder left out is left out everywhere. The naive comparison reads 10.54 against a truth of 6.23, and that +4.31 of leakage does not distribute itself evenly; it lands in whichever segments are most affected by season and tenure, which is the shape of a targeting rule built on nothing.
• Anything downstream of the treatment is the worst of them. Adding the support ticket takes the average to 2.59. A post-treatment column in a feature list is a targeting model conditioning on the future.

Every learner will produce some customers with negative predicted effects, and the two available mistakes are equal and opposite: naming a harmed segment that is only variance, and dismissing a real one as variance. Kestrelmark punishes the second. The sleeping dogs are 7.9 percent of the file at -8.87 each, and their individual effects are -10.0 or -7.0. A negative region here is a finding, and a model that smooths it away has lost the most valuable thing in the data.

• The first check is whether the negative region survives a change of learner. If the S-, T- and X-learners nominate different customers, there is no segment, only variance. If all three land on the same covariate values, that is about as much validation as this problem allows.
• The second check is whether a mechanism exists. Harm needs a story: an unsubscribe, an irritation, a cannibalised purchase. Here the story is in the segment definition itself, since these are customers who have already cancelled once and are being reminded of it.
• The third is sample size in the cell. Small cells produce extreme predictions in both directions, and the most extreme predictions in any uplift model are usually its least reliable ones. The smallest true segment is also the harmed one, which is the awkward combination.

So the discipline is not scepticism about negatives, it is the same discipline as everywhere else: a named group, a stated mechanism, and a number that holds up when the recipe changes.