The pooling bench

Twelve outlets, one voucher, one month. The generator draws each outlet's true redemption rate around 0.12 with genuine spread, then draws each customer's redemption from that rate. Footfall is deliberately unequal, from 18 vouchers at Northgate to 900 at Westfield, because that is what a real chain looks like and because the whole argument for pooling lives in the gap between those two numbers. The grand rate across all 2,443 vouchers is 0.141; the mean of the twelve true rates is 0.132.

Northgate is the recurring character. Its true rate is 0.192, the highest in the chain, and it redeemed one voucher in eighteen, an observed 0.056. Neither number is a mistake. One in eighteen is simply what a true 0.192 looks like at that volume about a fifth of the time, and the whole discipline exists for cases like it.

• The world. Twelve outlets from a fixed seed, identical for every learner, generated by the model written out in bayes-live.js where every rate is visible.
• Bench A. Exact conjugate Beta-Binomial posteriors. The interval is the 5.5 and 94.5 percent quantiles of 20,000 draws, an 89 percent equal-tailed interval, which is what ArviZ 1.x reports by default.
• Bench B. No pooling is each outlet's observed rate. Complete pooling is the grand rate for everybody. Partial pooling is an empirical-Bayes fit: the mu and kappa that maximise the marginal Beta-Binomial likelihood across the twelve outlets, searched on a grid, then each outlet shrunk toward mu with weight n over n plus kappa. It is a real fit, and it is the simpler of the two standard approaches; session 6 names the fully Bayesian version that puts a prior on kappa too.
• Bench C. One cumulative experiment: the first 50 customers per arm are the first 50 of the same 3,000. P(B > A) and both expected losses are Monte Carlo over 20,000 paired draws from the two posteriors.
• Nothing is stored. Every figure on the bench is computed when you click. A result that disagreed with the course notes would be the notes' problem.

Of the three ladder rungs, partial pooling wins on the Halloway data with a root-mean-square error of 0.025 against 0.031 for treating every outlet as identical and 0.054 for treating each as unrelated. The interesting number is not the winner, it is how the winner was chosen. Partial pooling learned from the twelve outlets that the right pooling strength was kappa = 116: each outlet's month counts for n over n plus 116, which is 0.13 at Northgate and 0.89 at Westfield. Nobody set that. The data did.

Change the world and the method changes its mind. In a world where the outlets are truly identical it learns a kappa at the top of the grid and pools completely, matching the grand-rate method to a few ten-thousandths. In a world where they are wildly different it learns kappa = 3.6 and barely pools, landing at 0.030 while complete pooling collapses to 0.145. That adaptiveness is the whole case for the method, and the bench lets you watch it happen rather than take it on faith.

Pooling is not free and the bench says so. On the real data, shrinking toward 0.135 improved six outlets and worsened six. The improvements are large and land where the data are thin: Northgate's error falls from 0.136 to 0.068, Hillcrest's from 0.088 to 0.021, Old Quay's from 0.058 to 0.023. The deteriorations are small and land where the data are thick: Market Row from 0.000 to 0.007, Parkside from 0.002 to 0.008, Docklands from 0.007 to 0.009, with The Arcade and Westfield unchanged to three decimals.

That asymmetry is the honest summary of shrinkage. It buys tenths on the outlets that need it and costs thousandths on the ones that do not, and the net is a lower error across the chain. It is also why the method is easy to sell badly: a manager at a large outlet will notice their number moved and will not notice it moved by less than the noise. Northgate, for its part, is still under-estimated at 0.124 against a truth of 0.192. The method halved the error. It did not remove it.