Bayesian A/B decisions

With flat Beta(1, 1) priors, each arm's posterior after the first fifty customers is exact and conjugate, session 1's arithmetic: five redemptions of fifty for A gives Beta(6, 46), ten of fifty for B gives Beta(11, 41). Draw 20,000 values from each. Pair them up. Every pair is one way the world could be, given what was seen, and every question about the decision is a question about the pairs.

P(B > A) is the share of pairs in which B's draw is higher: 0.915 at n = 50. Expected loss of choosing B is the mean over all pairs of max(theta_A - theta_B, 0): in the pairs where A is ahead, how far ahead, and zero everywhere else. At n = 50 that mean was 0.0029. The mirror, E[max(theta_B - theta_A, 0)], is the expected loss of choosing A, and it was 0.0984. Three numbers from the same two arrays, and only the losses are in the units of the business. Halloway's rule, written before the test, was to stop when the expected loss of the chosen arm fell below 0.001. At 0.0029, three times the threshold, it said keep going.

• P(B > A). The share of paired posterior draws in which B is ahead. A probability of superiority. Not a decision.
• Expected loss of choosing B. E[max(theta_A - theta_B, 0)], estimated as the mean of that quantity over the paired draws. In the units of the rate. The mirror is the expected loss of choosing A.
• Threshold. The expected loss below which Halloway agreed to act: 0.001. Set before the data, dated, and not moved.
• Stopping rule. Stop when the expected loss of the arm you would choose is below the threshold. Controls the cost of a wrong stop, not how often one happens.
• ROPE. Region of practical equivalence, Kruschke (2013). A band of differences too small to act on, compared against the posterior interval for the difference. The rule that can accept "no difference".
• 89 percent equal-tailed interval. The 5.5 and 94.5 percent quantiles of the draws. The ArviZ 1.x default, and the interval every ROPE comparison tonight uses.
• Wrong-stop rate. Across replays of the experiment from fresh seeds, the share that stopped on the arm that is truly worse. At Westfield, 2 of 200 at n = 100. A count, never a promise.
• Cumulative experiment. The n = 100 customers include the n = 50 customers. One test read at six points, not six tests.

Between n = 50 and n = 100 the observed rates did not move: 0.100 and 0.200 at both sizes. P(B > A) went from 0.915 to 0.975, a gain of six points. The expected loss of choosing B went from 0.0029 to 0.0005, a fall by a factor of six. Both come from the same left tail of the difference distribution, and the figure shows why they move at different speeds. The probability counts the tail's mass. The loss weights that mass by its distance from zero. When the posterior narrows, the tail loses mass and every remaining piece of it moves closer to zero, so the loss is hit twice.

That is the property that makes expected loss the right quantity to threshold. It is monotone in the evidence when a real difference exists, it is in the units of the stake, and it punishes a wide posterior harder than a probability does. The threshold itself, 0.001, is a business number: the average forfeit in redemption rate Halloway was willing to carry for a wrong call. A different business would write a different number, and the rule would still work the same way.

P(B > A) has a blind spot. Given enough data one arm is always ahead by a hair, so a probability of superiority can never say "these two are the same for any purpose that matters". Kruschke (2013), "Bayesian estimation supersedes the t test", Journal of Experimental Psychology: General, 142(2), 573-603, doi 10.1037/a0029146, supplies the missing rule. Decide, before the data, the band of differences too small to act on: the region of practical equivalence. Then compare the 89 percent equal-tailed interval for theta_B - theta_A to that band. Entirely outside the band: a difference worth acting on. Entirely inside: accept "no meaningful difference" as a positive finding. Straddling: not settled yet.

The second case is the one a t test cannot reach, and Kruschke's title is about exactly that. Estimation with a ROPE can conclude the null when the estimate is precise; a test can only fail to reject it. For Halloway the ROPE is a second rule beside expected loss, not a replacement. Expected loss says when acting is cheap enough. The ROPE says when the question is closed. The build-along runs the check with an example half-width of half a point, and the code prints its verdict rather than the page asserting one, because the width of the band is a business choice and not a course number.

Twenty thousand paired draws is an estimate of an integral, not the integral. P(B > A) to three decimals and expected loss to four will wobble in the last place between seeds. The canon values were computed once from fixed seeds and frozen, so your run should land on them to the precision printed and may differ by one in the last digit. What will not change is the verdict: 0.0029 is far enough above 0.001, and 0.0005 far enough below, that no seed flips the rule at either size.

The other thing the draws are not is a statement about the frequency of wrong decisions. They are a statement about one experiment's posterior. How often the rule, applied to many experiments, would stop on the wrong arm is a different quantity that needs many experiments, and Part 2 counts it.

Take the Westfield experiment, keep the true rates at 0.150 and 0.175, draw fresh customers from a fresh seed, and apply the same rule at n = 100. Do that two hundred times. In 23 of the 200 replays the rule would have stopped at n = 100. In 2 of the 200 it would have stopped on the wrong arm. That is a 1 percent wrong-stop rate at that sample size, and it is the number that belongs beside every stopped test: the rule is not free, it is cheap.

Which is why there is a claim you must not make. Never write that Bayesian A/B testing "has no peeking problem" or "lets you stop whenever you like". The expected-loss rule controls the cost of a wrong decision: each of those two wrong stops forfeited, on average, less than the threshold. It does not control the frequency of a wrong decision, and the 2 in 200 is the frequency. Whether reading the data as they arrive inflates that frequency, and by how much, is a real argument, and learn-experimentation-with-phoebe b6 owns it, along with Thompson sampling and bandits, the traffic-allocation cousins of tonight's rule that shift customers toward the winning arm while the test runs. Power and sample size are that course's b2, randomisation its b3. Tonight reports the measured rate and hands the argument over.

Halloway stopped at n = 100 and the ladder above is what it would have seen had it not. The observed gap shrank from ten points to 3.4 at n = 3000, closing on the true 2.5, while P(B > A) rose to 1.000 and the expected loss of choosing B stayed at 0.0000 to four decimals from n = 400 on. The decision was never in doubt after the second rung; what the later rungs bought was a better estimate of how much B wins by, not whether. The expected loss of choosing A tells the same story from the other side: 0.0984 at n = 50 down to 0.0339 at n = 3000, always the expensive option, falling as the posteriors narrowed onto a true difference smaller than the early data suggested.

A rule on expected loss stops when acting is cheap enough, and that is all it stops for. If the size of the lift matters for a budget, say so before the test and add a second condition, a ROPE or an interval width, so that the rule waits for the estimate as well as the decision.