The claim audit

When somebody says the email drove spend, they have asserted a picture: an arrow from email to spend, and crucially, the absence of anything else pointing into both. That second half is never stated and is always where the claim lives or dies. The person making it may not think of themselves as having drawn a diagram. They have, and the audit consists of drawing it for them and asking what is missing.

This is what makes the five questions portable. They do not depend on the method used, the tool, the sample size or the industry. They ask what the claimed picture is, what other pictures produce the same data, and whether anything was done to tell them apart. A claim that survives all five may still be wrong, but it will be wrong in a way that somebody has looked for.

• Ask about the claim, not the person. "What would we have to hold fixed for this comparison to be fair" is a question about the world. "Did you control for anything" is a question about their competence. The first gets answers.
• Ask in order and stop early. If the claim cannot survive question one, questions three through five are a waste of everybody's afternoon.
• Expect the answer to come from the business, not the analyst. What drives who gets treated is usually known by the campaign manager, the sales lead or the product owner. They are the ones to ask.
• Say what a pass looks like. An audit with no possible passing outcome is not an audit. Before you start, know what evidence would satisfy you, and say it.
• Write the finding as one sentence. Not a memo. The sentence you would send back to the author, which is the exercise in the second half of tonight.

Two separate rungs, and the whole audit turns on whether they match. The claim's rung is set by its verb: "is associated with" is rung 1, "drove" and "caused" and "delivered" are rung 2, "would not have happened otherwise" is rung 3. The evidence's rung is set by how the comparison was made: a difference between people who did and did not do a thing, however carefully sliced, is rung 1 unless something was actively done to make the two groups comparable.

A mismatch is not automatically fatal. It is the point at which the claim needs to justify the promotion, and questions three and four are how it does that. A match at rung 1 on both sides is fine and often exactly what was wanted.

• Campaign lift. "The email drove 10.54 units of incremental spend" is rung 2. The evidence is a comparison of emailed against not emailed, which is rung 1. Mismatch, so keep going.
• Feature launch. "Sessions rose 14 percent after we shipped the new home screen" is rung 1 as stated, and would be honest if it stopped there. Almost nobody stops there, and the next slide is titled impact.
• Retention. "Customers who use the mobile app retain better" is a clean rung-1 sentence right up until it becomes a roadmap item to drive app installs, at which point it has been read as rung 2 without anybody saying so.

Precisely enough that a colleague could disagree with the definition. This sounds pedantic and it is the question that most often ends an audit in ninety seconds, because a surprising number of claims dissolve when you ask what exactly was done to whom, and what exactly was measured afterwards.

• The treatment. Sent, or delivered, or opened? A single send or a three-week sequence? Everybody, or everybody the targeting rules selected? "Received the email" and "was chosen to receive the email" are different treatments with different answers.
• The outcome. Over what window, for whom, measured how? Spend in the following month is a different claim from spend in the following year, and both are different from spend among people who bought anything at all.
• The population. Everyone, or the 5,000 who were drifting? An effect on a selected subgroup does not transfer to the base, and the rollout decision is always about the base.

The Kestrelmark claim survives this one once it is pinned down: a single send, to 5,000 drifting subscription customers, with spend measured over the following quarter. That is a claim specific enough to be wrong, which is the compliment it deserves. A feature-launch claim usually does worse here, because "shipped the new home screen" was in truth a staged rollout to the users most likely to tolerate it.

The load-bearing question. Ask the room to name anything that makes somebody more likely to be treated and independently makes their outcome better. Two arrows from one thing, into both sides. Each one is a story in which the pattern appears exactly as reported and the treatment did nothing whatsoever.

The reason to ask this out loud, rather than to request it in writing, is that the answers come from operational knowledge rather than from the data. The person who set the targeting rules can list them in under a minute. The analyst who received the extract usually cannot, because the rules were never a column.

• Campaign lift. Winter: the campaign calendar sends more in December and people buy more in December. Tenure: long-standing accounts are on more lists and buy more anyway. Two candidates, both known to the business, together worth 4.31 of the reported 10.54.
• Feature launch. The rollout order. New surfaces go to the most engaged cohort first, so the feature's users were the heaviest users before it existed.
• Retention. Whatever makes somebody committed enough to install an app is also what makes them stay. Nobody has ever measured "committed", which is what makes this claim so durable and so hard to act on.

Ask for the covariate list. Not the model, not the code: the list of things held fixed. Then read it for one thing before anything else. Is any item on that list something the treatment itself could have changed?

Holding fixed something that the treatment caused removes part of the effect you were trying to measure, and it can do worse than that. On Kestrelmark's data, holding fixed whether the customer opened the email moves the estimate to 5.48, because opening is how the email works. Holding fixed whether they raised a support ticket, which happens after the email and after the spending, moves it to 2.59, and that one is by far the most damaging single item anybody has put on this covariate list. Both of them look like careful analysis and both are worse than the correct set's 6.56.

• The green flag. A short list, every item of which was determined before the treatment, and each of which somebody can explain the role of.
• The red flag. A long list assembled because it improved the fit. Session 3 shows what each kind of wrong variable costs, and the harmful ones do not cancel out against each other.
• The subtle one. "It was measured before the treatment" is not by itself a safety argument. Kestrelmark's loyalty score predates the send and holding it fixed still moves the estimate to 5.26, which puts it in the middle of the three harmful adjustments rather than at the mild end. Session 3 explains why.

Put the burden where it belongs: on the claim, not on the sceptic. A strong claim can name the thing that would overturn it. "If tenure were driving this, we would see the effect vanish among accounts of the same age, and it does not" is an answer. So is "an unmeasured factor would have to be about as strong as the season effect, which we think is implausible, and here is why".

An answer of "nothing could really show this is wrong" is not a strength. It is the finding, and it should be reported as one. The follow-up worth asking is the cheapest version of the test: what would we have to look at, and could we look at it this week?

• Campaign lift. Compare within the same month and the same tenure band. If the gap collapses, season and tenure were carrying it.
• Feature launch. Look at what the adopting cohort was doing in the weeks before the feature existed. If they were already ahead, the rollout order is the story.
• Retention. Ask what a randomised nudge to install would cost. If nobody will fund it, that is a real answer about how much the claim is trusted.

Certain words carry a rung-2 promise, and when they are attached to an observational comparison, the promise was not paid for. This is rarely deception. Writing "customers who received the email subsequently spent 10.54 more, and we have not established why" is accurate and nobody puts it on a slide, so the sentence drifts upward one word at a time between the analysis and the review.

• Drove, led to, delivered, generated. All rung 2. Each asserts that acting produced the outcome.
• Incremental. The strongest of them, and the most casually used. Incremental means "over and above what would have happened otherwise", which is a claim about a world that did not occur.
• Impact, uplift, contribution. Rung 2 in a noun's clothing, which is why they slip past. "Campaign impact: 10.54" reads like a measurement and asserts a cause.
• The honest forms. "Associated with", "correlated with", "customers who did X also did Y", "we observed". These are rung 1 and they are not weaker writing. They are accurate writing, and a leader who visibly rewards them will get more of them.

The reaction to a tell is a question, never a conclusion. The claim may be perfectly well supported and the author may simply have written naturally. What the word does is tell you which of the five questions to ask first.

When a claim comes with a list of what was controlled for, read it as a document in its own right. It tells you what the analyst believed the picture looked like, and its shape is informative even before you know what the variables mean.

• A list with anything measured after the treatment is a warning. Opens, clicks, tickets, subsequent visits, downstream status. If the treatment could have changed it, holding it fixed changes the question being answered and usually damages the estimate. The reason is session 3, and the numbers are unambiguous: 5.48 for the mediator and 2.59 for the later collider, against 6.56 for the correct set.
• A very long list is a warning of a different kind. It usually means variables were added until the fit improved. Model fit is not evidence about causation, and the harmful additions on Kestrelmark's data push the same direction and stack to a bias of 5.25 rather than cancelling each other, which is a worse answer than holding nothing fixed at all and its bias of 4.31.
• An empty list is not automatically bad. If the treatment was genuinely randomly assigned, nothing needs holding fixed. Ask whether it was, and how.
• The item you cannot get a role for is the item to worry about. Ask what job each variable is doing. Anything that gets "it was in the table" as an answer is there by habit.

Two failure shapes that are older than any of the vocabulary and still account for a large share of confident wrong claims.

• The missing denominator. "Four hundred customers who received the offer upgraded." Out of how many? A number without its base is not a rate, and a rate is the minimum needed to compare anything. The same trick appears as counts of successes with no mention of attempts, and as testimonials with no mention of the people who did not respond.
• The self-selected group. The people who opted in, installed the app, attended the webinar or joined the loyalty programme chose to. Whatever made them choose is very often the same thing that makes their outcome good. This is why "customers who use X retain better" is nearly always true and nearly never actionable: the comparison group differs by the one thing you cannot see.
• The survivor version. Measuring outcomes only among people still present at the end. If the treatment affects who is still there, the surviving group is not the group you started with, and the comparison silently changed populations.

The reason to keep these on the list alongside the five questions is that they are visible in the claim's own wording. You do not need the data to notice that a denominator is absent or that a group joined itself.

This is the honest limitation and it deserves saying plainly, because otherwise the method quietly rewards the wrong behaviour. Every one of the five questions works by taking the claim seriously enough to test it. A claim that says "the campaign contributed to a strong quarter" cannot fail question two, because there is no treatment definition to be wrong about. It cannot fail question three, because there is no comparison to confound. It sails through, and it means nothing.

• Vagueness is not strength. A claim that cannot be wrong is not a claim that survived scrutiny; it is a claim that could not be scrutinised. Record that as the finding, in exactly those words.
• The correct response is a request, not a rejection. Ask for the treatment, the outcome, the window and the population. If the answer is that those were never defined, the audit is complete and the answer is that there is no result yet.
• Watch for vagueness arriving under pressure. A specific claim that becomes vague after being questioned has told you something about how much its author believes it.
• Reward the specific version publicly. The team that writes "a single send to 5,000 drifting customers, spend measured over the following quarter, 10.54 unadjusted" has made itself easier to challenge. If challenging is punished, the next version will be a sentence about contribution.

When the audit concludes that the claim needs an actual estimation method rather than a sharper sentence, that is where learn-experimentation-with-phoebe begins: potential outcomes, propensity scores, difference-in-differences and the rest are covered there in working depth.