Reading a posterior

Session 2 left you with the idea that an estimate from small data is a distribution: a spread of plausible values for the rate, heavier where the evidence points and thinner at the edges. Tonight the useful picture is more concrete. Think of the posterior as twenty thousand plausible values written on twenty thousand slips of paper. Each slip is one way the world could be, given what you believed and what you saw. Every question you could ask about the rate becomes a question about the slips.

With two vouchers there are two piles. Draw one slip from A's pile and one from B's and compare them. Do it twenty thousand times. The share of draws in which B's slip is higher is P(B > A), the probability that B is the better voucher. At Westfield after fifty customers per arm it was 0.915. The same draws answer a second question: in the draws where B is actually worse, how much worse is it, on average, counted across all the draws? That is the expected loss of choosing B, and at n = 50 it was 0.0029. The first number tells you how likely you are to be wrong. The second tells you what being wrong costs. Only one of them is a decision.

• Posterior. The distribution of plausible values for a rate after the data. Tonight, twenty thousand draws from it, one pile per voucher.
• P(B > A). The share of paired draws in which B's rate beats A's. A probability, not a decision.
• Expected loss. How much redemption rate you forfeit on average if you pick this arm and it turns out to be the worse one. A probability times a stake, so it is a decision.
• Threshold. The expected loss below which Halloway agreed to act: 0.001. Chosen before the test, written down, and not moved during it.
• Stopping rule. Stop when the expected loss of the arm you would choose is below the threshold. It controls the cost of a wrong decision, not how often one happens.
• 89 percent interval. The range holding 89 percent of the posterior, cut equally from both tails, the 5.5 and 94.5 percent points. Every interval tonight is this kind and says so.
• Credible interval. The Bayesian interval. It is the sentence most people think a confidence interval is: there is an 89 percent probability the rate is in here. The frequentist sentence is different, and learn-statistics-with-phoebe owns it.
• ROPE. A region of practical equivalence: the band of differences too small to care about. Named by Kruschke (2013), and the device that lets a posterior say "no meaningful difference" out loud.

At n = 50 the two piles of slips overlap a great deal. In 91.5 percent of paired draws B's slip is higher, which sounds decisive, but look at the other 8.5 percent. In those draws A is the better voucher, and in some of them it is better by a lot, because fifty customers cannot pin either rate down. Average the shortfall across every draw, counting zero for the draws where B wins, and you get 0.0029: on average, choosing B now forfeits 0.0029 of redemption rate to the possibility that B is the worse arm. Halloway's threshold was 0.001. The rule said keep going, and the reason was not that 0.915 fell short of some magic cut-off. The reason was that being wrong was still expensive.

Fifty customers later, with the observed rates unchanged at 0.100 and 0.200, the piles have narrowed. P(B > A) rose to 0.975, a modest move. The expected loss of choosing B fell to 0.0005, a move by a factor of six, because the draws where A wins are now both rarer and closer. That is the general shape: the probability tells you how often you would be wrong, the expected loss tells you how much it would cost, and a threshold on the cost is what makes it a rule you can defend afterwards. The other side of the ledger is worth reading too. The expected loss of choosing A was 0.0984 at n = 50 and 0.0989 at n = 100. Nobody in the room was proposing A, but the number says why: staying with the current voucher was the expensive option all along.

The sentence is: "If we choose B and B is actually the worse voucher, we forfeit on average this much redemption rate." It is a probability multiplied by a stake, summed over every way the world could be, and that is exactly the shape of every other expected-cost number a finance team already uses. Nothing about it is novel except that the probabilities come from a posterior rather than a spreadsheet assumption.

Two properties make it the right number to put a threshold on. It falls as data arrive, because the losing draws get rarer and smaller together, so a threshold on it will eventually be met if a real difference exists. And it is in the units of the thing you care about, redemption rate, so the threshold can be set by asking what forfeit Halloway is willing to carry, rather than by asking what probability feels comfortable. Halloway said 0.001, a tenth of a point of redemption rate. A different business would say a different number, and that is the point: the threshold is a business decision made in advance, and the posterior only reports whether it has been reached.

Every interval tonight is an 89 percent equal-tailed credible interval: the 5.5 and 94.5 percent points of the posterior. It supports the sentence "there is an 89 percent probability the rate is in this range", which is the sentence almost everybody hears when a confidence interval is read out in a meeting. The frequentist confidence interval supports a different sentence, about the procedure rather than the rate, and learn-statistics-with-phoebe session b7 owns that sentence. It is not re-derived here. What matters for reading a posterior is only this: when a Bayesian analyst says 89 percent, they mean it about the rate in front of you, and you are allowed to use it that way.

Replay the Westfield experiment two hundred times with the same true rates and different customers, apply the rule at n = 100 each time, and 23 of the 200 would have stopped there. Of those 200 replays, 2 would have stopped on the wrong arm. State it as a count, because that is what it is. The rule kept the cost of each wrong stop small, and it did not make wrong stops impossible. Nothing does.

Which is why there is a sentence you should never hear from an analyst and never repeat: that Bayesian A/B testing "has no peeking problem" or "lets you stop whenever you like". The expected-loss rule controls the cost of a wrong decision. It does not control the frequency of one, and the 2 in 200 above is the frequency. Whether a stopping procedure applied as data arrive inflates that frequency, and by how much, is a real argument, and learn-experimentation-with-phoebe session b6 owns it. Tonight's job is only to report the measured rate and to make sure nobody in the room mistakes a cheap rule for a free one.

P(B > A) can only ever say which arm is ahead. It cannot say "these two are the same for any purpose that matters", because with enough data one arm is always ahead by a hair. Kruschke (2013), "Bayesian estimation supersedes the t test", Journal of Experimental Psychology: General, 142(2), 573-603, supplies the missing move. Decide in advance the band of differences too small to care about, the region of practical equivalence, and compare the posterior interval for the difference to it. If the interval falls entirely inside the band, you can accept "no meaningful difference" as a positive finding, something a test cannot do. If it falls entirely outside, the difference is real and large enough to act on. If it straddles, you do not know yet.

For a leader the useful part is the first case. A precise estimate that lands inside the ROPE is a licence to stop spending on a question, and it is the one outcome that a probability-of-difference alone can never deliver. Ask for the band before the data arrive, in the units of the decision: at Halloway, "a difference under half a point of redemption rate does not change what we do" would have been enough.

Whatever is on the screen, three questions read it. Where is the mass? The bulk of the plausible values, the region the posterior mean and the middle of the interval both point at. How wide is it? The 89 percent equal-tailed interval, and whether its two ends would lead you to different actions. What does the width cost me if I act now? The expected loss of the action you are about to take, which is the width translated into the stake. The first two are about the estimate. The third is the decision, and it is the one the meeting is actually there to make.