Campaign lift and brand health

Before-and-after compares this month with last month. Everything that changed between them, the season, a competitor's price cut, a newspaper mention, the weather for hot drinks, lands in the difference and gets credited to the campaign. A holdout runs both groups through the same weeks, so all of that hits both sides equally and cancels out.

Platform attribution counts conversions from people who saw or clicked an ad, inside a window and under a model the platform chose. Many of those people would have bought anyway: loyal subscribers, people already searching for the brand. Attribution has no way to subtract them, because it never sees a comparable group that was not shown the ad. A holdout is that group.

The holdout's honesty has a price: the held-out customers really do get nothing for the length of the test, and if the campaign works, some sales are given up to learn that it works. That trade is a marketing decision, made by the marketing lead with the data team naming the size of the group and the length of the test. Designing the size properly, the power and sample-size side, is taught in Experimentation ↗.

For the wider discipline, the standard reference is Ron Kohavi, Diane Tang, Ya Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020). Its publisher's description sums up the whole field in one line: "getting numbers you can trust is hard". Only that description was read for this page, not the chapters, so no specific holdout claim here is attributed to the book.

A Kettle & Co Meta campaign (constructed) reports £98,400 of revenue on £24,000 of spend: a platform ROAS of 4.1. The marketing lead's question is whether to double the budget. A conversion lift test on the same campaign finds that 38 percent of the reported conversions were incremental, meaning they would not have happened without the ads. Incremental revenue is then £37,392, and incremental ROAS is 1.56.

Both numbers are "true" in their own terms. 4.1 is what the platform credited under its own window and model; 1.56 is what the ads caused. Only the second answers "should I spend more?", and it is a very different answer: a channel returning £1.56 per pound before margin may not be paying for itself at all. Whether it does is a finance and margin question (Angles session 5), and whether to double is still the marketing lead's call. What the data team owes her is the right number to make it on.

One more honesty note: the 38 percent is itself an estimate from a test, with its own uncertainty. Ask for the interval on the lift figure, not just the point.

Both big ad platforms offer their own lift tests. As of October 2026, read at source on 10 October 2026; check each platform's current help page before you plan one, because these pages change.

Meta Conversion Lift (Meta Business Help Center, "About Conversion Lift"). It splits people into randomised test and control groups, the control being the people who don't see your ads, and uses an intent-to-treat approach: people are counted in the group they were assigned to, whether or not an ad actually reached them. Lift is the difference in conversions caused by the presence of the ads, and Meta says the lift figure is estimated. Its eligibility guide is around USD 5,000 of spend and 500 conversions, and the test comes at no additional cost.

Google Ads Conversion Lift (Google Ads Help, "About Conversion Lift"). The control group is again people who don't see your ads, split either user-based or geography-based. User-based tests rely mostly on opt-in traffic, and the feature is not available for all accounts.

A platform's lift test is far better evidence than the same platform's attribution report, because it has a real control group. It is still run, analysed and reported by the seller of the ads. A data team should read its method notes, ask for the interval, and where the decision is big, compare it with a holdout the company controls (an email or CRM holdout, or a geography test designed with the Attribution team).

The interval says the true effect of the emails over these four weeks is somewhere between a small loss of 0.014 orders per customer and a gain of 0.028. Zero is inside that range, so the test cannot rule out no effect. But a gain of 0.02 is inside it too, and across 18,000 customers that would be a real number of orders. The test did not find that email does nothing. It found that, at this size, it cannot tell.

The honest sentence names both halves: what the point estimate is, and that it cannot be distinguished from zero at this test size, with the interval written out. Then it names what would settle it. The marketing lead can decide to keep the cadence, cut it, or run a bigger test; the data team's job is to make sure she knows which of those the evidence actually supports, which tonight is none of them strongly.

The interval is wide mostly because the held-out group is small. 18,000 treated customers pin down their own rate quite tightly; 2,000 held-out customers do not, and the difference can only be as precise as its least precise side. A 10 percent holdout is a common choice because it gives up few sales, and the price of that choice is exactly this width.

Two levers narrow it. A bigger held-out group: interval width shrinks with the square root of the group size, so halving the width needs roughly four times as many held-out customers. A longer test: more weeks means more orders per customer to count, which also tightens the interval, as long as the effect does not fade over the period. Both cost something real: more customers given nothing, or more weeks before a decision. How large and how long, worked out properly before the test starts, is the power calculation taught in Experimentation ↗. The analyst's job in the readout is to name the lever and the cost in a sentence, and let the marketing lead decide whether the question is worth it.

Calling it a win on the point estimate. "Email lifted orders 3.4 percent." True of the dot, false of the evidence. If the next test comes back at -0.005, the data team has spent its credibility on noise.

Calling it a loss. "The test proved email doesn't work, cut the sends." The interval allows a gain of 0.028 per customer. Cutting a channel on a test too small to see it is a decision made on absence of evidence, presented as evidence of absence.

Quietly not reporting it. The most common one, and the most damaging. A team that only reports tests that "worked" teaches the business that tests always work, and every future readout is discounted. A null result, written up in five lines, is how a data team earns the right to be believed when a result is real.

Each wave asks 500 different people. Even if Kettle & Co's true awareness had not moved at all, two samples of 500 would rarely land on exactly the same percentage; a few points of difference is what sampling alone produces. The interval on the change, -2.9 to +8.9 points, says the true change could be a small fall, nothing, or a large rise. "+3 points" in a board pack, with no interval, reads as growth; with the interval, it reads as "not yet distinguishable".

Two other things move tracker numbers that have nothing to do with the brand: changes in who answers (a different panel mix, a different season) and changes in the questionnaire (a new brand added to the list, a question moved). Ask the agency, which owns the fieldwork, whether either changed between waves before reading any move at all.

Brand effects are slow and broad, so the tracker's job is to show a direction over many waves, not a verdict between two. Five or six waves drifting the same way, each with its interval drawn, is much stronger evidence than any single pair, even when every pair on its own crosses zero. Practical habits: always plot every wave, never just the latest two; draw the interval on each point; and when a single wave jumps, wait for the next one before writing it up.

Running the tracker itself, the questionnaire, the panel, the fieldwork, is brand's and the agency's craft, and brand owns what the tracker is for. The data team is consulted: it brings the interval, the trend line, and the join to internal signals that move on the same slow clock, such as branded search volume. Data informs the bet; the business places it.