Borrowing strength

Twelve outlets, twelve observed rates, one decision about how to read them. The first option is to treat each outlet alone: Northgate is 0.056, Westfield is 0.141, and the fact that they belong to the same chain is ignored. This is what the monthly pack does by default, and it scores 0.054. The second option is the opposite: treat the chain as one, give every outlet the grand rate of 0.141, and ignore the fact that outlets differ. It scores 0.031, better than the first, because most of the twelve observed rates are further from their own truth than the chain average is. That is a fact about small samples, not about the outlets.

The third option sits between them and is the subject of tonight. Each outlet's estimate is a blend of its own month and the chain, and the blend leans toward the chain for outlets with little data and toward the outlet's own month for outlets with a lot. It scores 0.025. The name for it is partial pooling, the eleven informing the twelfth, and its whole character is in one question: how much should one outlet's month be trusted? Under the first option the answer is completely; under the second, not at all; under the third, it depends on how big the month was and how different the outlets really are, and both of those can be measured.

• No pooling. Each outlet estimated from its own data alone. Twelve separate problems. RMSE 0.054 on the Halloway data.
• Complete pooling. Every outlet given the chain-wide rate. One problem, twelve copies of the answer. RMSE 0.031.
• Partial pooling. Each outlet blended with the chain, the blend set by its sample size and by how different the outlets are. RMSE 0.025.
• Shrinkage. What partial pooling does to an estimate: pulls it from the observed rate toward the chain centre. Thin outlets are pulled a long way, thick ones barely.
• mu. The centre the estimates are pulled toward. Learned from the twelve outlets: 0.135.
• kappa. How hard they are pulled, expressed as a pseudo-count, the same currency as session 1. Learned from the twelve outlets: 116. An outlet's own month carries weight n / (n + 116).
• Hierarchical model. The general name for a model in which the outlets are treated as draws from a chain-level distribution whose centre and spread are themselves estimated. Partial pooling is what it does for you; practitioner session 6 builds one.
• RMSE. Root-mean-square error of the twelve estimates against the twelve true rates. The score every rung of the ladder is judged on, possible only because the truth was written into the data.

Session 1 gave you the currency: a prior is worth a number of pseudo-observations, and an outlet's own data carry weight n over n plus that number. Tonight's method uses exactly that arithmetic, with one difference that changes everything. The prior's centre and its pseudo-count are not chosen by an analyst. They are learned from the twelve outlets themselves, by asking which chain-level centre and which spread of outlets around it make the twelve observed months most plausible. On the Halloway data the answer is a centre of mu = 0.135 and a pseudo-count of kappa = 116.

Every outlet then gets the same treatment. Its own month carries weight n / (n + 116); the chain centre carries the rest. At Northgate that is 18 / 134, which is 0.13: the outlet's own month is trusted thirteen percent, and the other eleven outlets supply the remaining eighty-seven. At Westfield it is 900 / 1016, which is 0.89: the flagship's own month is trusted almost entirely, because it has earned it. Nobody set 0.13 or 0.89. Nobody set 116. The question "how much should one outlet's month be trusted" was answered by measuring how much the outlets actually differ, and the same measurement would have produced a different answer on a different chain.

Northgate's observed 0.056 carries weight 0.13; the chain centre of 0.135 carries 0.87; the blend is 0.124. Its error against the true 0.192 falls from 0.136 to 0.068. That is the number that took it off the closure list, and it is worth being precise about why. The method did not discover that Northgate is the best outlet in the chain. It could not: one redemption in eighteen does not contain that information, and 0.124 is still well below 0.192. What the method did was refuse to believe that an eighteen-voucher month could place an outlet at 0.056 when eleven neighbours running the same voucher sat around 0.135. It halved the error. It did not remove it.

That is the right way to describe what shrinkage buys a leader. It does not find the truth from thin data. It stops thin data from producing confident nonsense, by lending the thin outlet the chain's experience in proportion to how thin it is. Northgate stayed open not because the analysis proved it was good but because the analysis showed the case for closing it was made of noise.

Put tonight's learned prior next to session 1's asserted one. The asserted prior was Beta(12, 88): centre 0.120, worth 100 pseudo-observations, and it put Northgate at 0.110 with an interval that missed. The learned prior has centre 0.135 and is worth 116 pseudo-observations, a similar strength and a slightly higher centre, and it puts Northgate at 0.124. The difference between them is not mainly in the numbers. It is in where the numbers came from. The asserted 100 was a feeling about the chain's history; the learned 116 is a measurement of how much the twelve outlets differ from each other this month, and it would fall if they differed more.

And the learned prior still paid session 1's price at Northgate. Northgate really is unlike the chain, the one outlet whose truth sits far above the centre, and a prior worth 116 pseudo-observations pulled it toward the crowd anyway. The estimate of 0.124 is better than 0.056 and worse than the truth by 0.068. There is no version of borrowing strength that is free for the outlet that genuinely differs; the method's virtue is that it charges the thin outlets a fee proportional to their thinness and charges the thick ones almost nothing, and that the fee is set by the data rather than by whoever built the slide. Session 5 takes up the argument about whether a prior should be learned this way at all, and does not settle it.

A ladder score of 0.025 against 0.054 is an average, and averages hide who paid. Outlet by outlet, partial pooling made six estimates better and six worse. The helps were large and landed on the thin outlets: Northgate's error fell from 0.136 to 0.068, Hillcrest's from 0.088 to 0.021, Old Quay's from 0.058 to 0.023. The hurts were small and landed on the thick ones: Market Row's error rose from 0.000 to 0.007, Parkside's from 0.002 to 0.008, Docklands' from 0.007 to 0.009, and The Arcade and Westfield did not change to three decimals. Shrinkage buys tenths on the outlets that need it and costs thousandths on the ones that do not.

Say both halves. A slide that shows only the six helps is selling something; a manager who has only heard about the six hurts will veto the method and be wrong to. The trade is asymmetric in a way that favours the chain as a whole, and it is asymmetric for a reason you can explain: the thick outlets barely moved because the method trusted their own months, at weights near 0.89, and the thin ones moved a lot because the method did not trust theirs, at weights near 0.13. The hurts are the price of the trust; the helps are the price of the doubt. Northgate, the biggest winner, is still under-estimated at 0.124 against 0.192.

In the world where the twelve outlets are truly identical, complete pooling is the right answer by construction, and it scores 0.0065. Partial pooling learns that the outlets are the same, pushes its pseudo-count to the ceiling of what it was allowed to consider, pools almost completely, and scores 0.0051, a hair better still. In the world where the outlets are wildly different, complete pooling is the wrong answer by construction and collapses to 0.145, five times worse than doing nothing. Partial pooling learns a pseudo-count of 3.6, barely pools at all, and scores 0.030, level with treating each outlet alone.

That is the property a leader is buying. It is not that partial pooling is the best method in every world; in the identical world complete pooling is nearly as good, and in the wild world no pooling is. It is that partial pooling is never the wrong method, because it asks the data how alike the units are before deciding how much to borrow. The analyst who chooses complete pooling is asserting that the outlets are alike; the one who chooses no pooling is asserting that they are not; the one who chooses partial pooling is declining to assert either and measuring instead.

The objection will come from the thick outlets, and it deserves a straight answer rather than a lecture. Yes, the number moved. At Market Row the error went from 0.000 to 0.007; at Parkside from 0.002 to 0.008; at Docklands from 0.007 to 0.009. Those are the movements. Now put them beside what session 2 showed about the same outlets: Westfield's own 89 percent interval, at 900 vouchers, was 0.037 wide, and Westfield did not move to three decimals. At a big outlet the estimate moved by less than the noise already in it, and the manager's month was never known to the precision the movement disturbs.

The honest version of the answer has three parts. The number moved by a few thousandths, which is less than the month-to-month noise at your volume. It moved because your outlet's month was trusted at nearly full weight, 0.89 at the flagship, and lent a little of its stability to outlets that had none. And in return, the outlets you are compared against are no longer swinging by tenths on eighteen vouchers, which makes the ranking you are judged on fairer to you as well. Nobody's number is now a fiction; every number now says how much it can be trusted. What that does to a monthly review is session 6's subject, and what it does to a comparison between two arms is session 4's.