Hierarchical models

Three ways to estimate twelve rates. No pooling: each outlet's observed rate, y over n, twelve unrelated problems. Complete pooling: one rate for all, the grand redemption rate of 0.141, twelve copies of one answer. Partial pooling: each outlet shrunk toward a centre learned from all twelve, by an amount that depends on its own volume and on how different the outlets really are. The score is the same for all three: root-mean-square error of the twelve estimates against the twelve true rates, which can be computed because the true rates are in the generator.

On the Halloway data the ladder reads 0.054, 0.031, 0.025. The order is what McElreath's lectures 12 and 13 predict and the numbers are what the arithmetic gives. Complete pooling beats no pooling because most small outlets' observed rates are further from their own truths than the chain centre is; partial pooling beats both because it trusts each outlet's month in proportion to how much month there was. Tonight is about how that proportion is found, what it is worth, and what it costs.

• mu. The chain-level centre every outlet is shrunk toward. Learned from the twelve: 0.135.
• kappa. The pooling strength, expressed as a pseudo-count in session 2's currency. Learned from the twelve: 116. A Beta(mu kappa, (1 - mu) kappa) prior on each outlet's rate is worth kappa pseudo-observations.
• Weight. An outlet's own month carries n / (n + kappa) of its estimate; the centre carries the rest. 0.13 at Northgate, 0.89 at Westfield.
• Marginal Beta-Binomial likelihood. The probability of an outlet's observed count given mu and kappa, with the outlet's own rate integrated out. Summed in logs across the twelve, it is the function the grid search maximises.
• Empirical Bayes. Fixing mu and kappa at the values that maximise that likelihood, then treating them as known. Fast, exact, and honest about being the simpler method.
• Hierarchical model. The fully Bayesian version: a prior on mu and kappa too, and a posterior over them alongside the twelve rates. What the PyMC block adds.
• Shrinkage. What partial pooling does to an estimate: pulls it from the observed rate toward mu. Thin outlets a long way, thick ones barely.
• RMSE. Root-mean-square error of twelve estimates against twelve true rates. Possible here and almost nowhere else.

Put a Beta(mu kappa, (1 - mu) kappa) prior on every outlet's rate. That is session 2's currency exactly: a prior centred on mu and worth kappa pseudo-observations. Now ask, for a candidate pair (mu, kappa), how probable the twelve observed counts are with each outlet's own rate integrated out. That probability has a closed form, the Beta-Binomial, and its logarithm is two calls to betaln per outlet. Sum it across the twelve, do that for every pair on a grid, and keep the pair with the highest total. On the Halloway data the winner is mu = 0.135, kappa = 116.

Everything else follows without a further decision. Each outlet's shrunk estimate is the posterior mean under that prior, which is the observed rate weighted by n / (n + 116) plus the centre weighted by the rest. At Northgate 18 / 134 is 0.13; at Westfield 900 / 1016 is 0.89. Nobody chose 0.13, 0.89 or 116. The grid search chose them by asking which chain-level description made this month's twelve counts most likely, and a different chain, or a different month, would have produced a different kappa. This is what "the method reads the answer off the data" means in practice, and the build-along is where you watch it do so.

The RMSE fell from 0.054 to 0.025, and averages hide who paid. Outlet by outlet, six errors fell and six rose. The falls are large and land on the thin outlets: Northgate 0.136 to 0.068, Hillcrest 0.088 to 0.021, Old Quay 0.058 to 0.023, Kingsway 0.046 to 0.027, Riverside 0.043 to 0.017, Abbey Green 0.024 to 0.010. The rises are small and land on the well-measured ones: Market Row 0.000 to 0.007, Parkside 0.002 to 0.008, Station 0.020 to 0.024, Docklands 0.007 to 0.009, and The Arcade and Westfield did not change to three decimals. Shrinkage buys tenths on the outlets that need it and costs thousandths on the ones that do not.

Say both halves, always. The asymmetry is not luck; it is the weights. The thick outlets barely moved because the fit trusted their own months, at weights near 0.89, so the centre had little room to pull. The thin ones moved a lot because the fit did not trust theirs, at weights near 0.13, and lent them the chain's experience instead. Market Row is the sharpest case: its observed rate happened to sit on its truth to three decimals, and shrinkage moved it off by 0.007. That is a real cost, and it is the price of a method that cannot know in advance which thin-looking month happened to be right. Northgate, the biggest winner, is still under-estimated at 0.124 against 0.192. The method halved the error. It did not remove it.

Session 2's informed prior was Beta(12, 88): centre 0.120, worth 100 pseudo-observations, chosen from a feeling about the chain's history. Tonight's learned prior has centre 0.135 and is worth 116, similar strength, slightly higher centre. At Northgate the asserted prior gave 0.110 and the learned one gives 0.124. The improvement is real but modest, and it is not the point. The point is provenance: 100 was asserted and 116 was measured, and a measured pseudo-count comes with a reason anyone can check, namely that it is the value under which the twelve observed months are most plausible together.

The learned prior still paid session 2's price at Northgate, because Northgate genuinely differs from the chain and 116 pseudo-observations of chain experience pulled it toward the crowd anyway. There is no version of pooling that is free for the unit that truly stands apart. What the data-chosen kappa buys is that the fee is set by how alike the units actually are, and that in a world where they were not alike, it would fall. Part 2 shows it doing exactly that. Whether a prior ought to be learned this way at all is the argument of leader session 5, and it is not settled.

In the world where the twelve outlets are truly identical, complete pooling is right by construction and scores 0.0065. The fit sees that the twelve counts are all consistent with one rate, pushes kappa to the ceiling of its grid, pools almost everything, and scores 0.0051. In the world where the outlets are wildly different, complete pooling is wrong by construction and collapses to 0.145, worse than doing nothing by a factor of more than four. The fit learns kappa = 3.6, barely pools, and scores 0.030, a hair better than the 0.033 of no pooling. On the real data it learned 116 and scored 0.025.

Read the kappa column as the method's opinion about how alike the units are, formed from their counts and nothing else. At 3.6, Northgate's own month would carry 18 / (18 + 3.6), about 0.83, instead of 0.13: almost all of the trust goes back to the outlet, because the chain has stopped being informative about it. That is the whole case for the approach. Not that partial pooling is the best estimator in every world, since complete pooling is nearly as good in the identical one and no pooling is close in the wild one, but that it is never the wrong estimator, because it measures before it pools.

The twenty-line fit finds the single best (mu, kappa) and then treats it as known. Every shrunk estimate and every weight is conditional on kappa being exactly 116, and the fit has no way to say how sure it is of that. A hierarchical model removes the assumption. Put a prior on mu and a prior on kappa, put the same Beta(mu kappa, (1 - mu) kappa) prior on each outlet's rate, and let PyMC sample all fourteen quantities together. What comes back is a posterior over mu and kappa themselves, alongside the twelve rates, so the pooling strength carries its own uncertainty and each outlet's interval is wider by however unsure the model is about how much to pool.

That is the one thing the grid cannot give you, and it is why the PyMC block in the build-along has the same structure with two more lines. The bench in session 10 uses the empirical-Bayes fit because it is exact, fast and honest about being the simpler method; the hierarchical model is what you reach for when the question "how sure are we about kappa" is going to be asked, which in a room deciding an outlet's future it will be. Read the diagnostics with particular care here: a model with a level above the units is exactly where session 4's checks earn their keep.

Outlets in a chain, players in a league, regions under one campaign, cohorts of one product: any time many small units are measured the same way and could reasonably resemble each other, the structure tonight applies unchanged, and the question is always the same one, how much should one unit's own record be trusted. learn-ai-sports-with-phoebe b3 names Bayesian hierarchical models that pool player effects with shrinkage as what works for expected goals, and this is the session that page points at: a player with few shots is Northgate, a player with hundreds is Westfield, and the league's kappa is learned from all of them.

Two things the structure does not do. It does not explain why the units differ; why Northgate's true rate is 0.192 and Old Quay's is 0.090 is a causal question, and learn-causal-inference-with-phoebe owns it. And it does not tell you whether the units should be compared at all. That is a judgement about the family, made before any fitting, and the fit will pool whatever you hand it.