Checking the model

Every model from sessions 5 to 7 is a machine for turning a rate into a count: a Beta on the rate, a Binomial on the redemptions. Run the machine forwards instead of backwards and it produces data. Run it forwards from the prior, before touching the observed counts, and you get the prior predictive: the redemption counts the prior alone considers plausible. Run it forwards from the posterior, after the fit, and you get the posterior predictive: the counts the fitted model expects in a fresh month. Both are simulations, both are cheap, and both end with the same question. Does this look like data that could exist, and does it look like the data that did?

The prior predictive is the check leader session 5 lands on as common ground, because it needs no agreement about where priors come from. A subjectivist, a reference-prior analyst and a weakly-informative one can all look at what Beta(40, 60) predicts for 900 vouchers and agree it is not a bakery chain. The posterior predictive is McElreath's lecture 7 tool for asking whether the fitted model has missed a shape in the data. Neither check produces a verdict on its own. They produce a picture, and the analyst produces the verdict.

• Prior predictive check. Simulate data from the prior alone, with no observed counts used, and ask whether it looks like data that could exist. pm.sample_prior_predictive() in PyMC 6; its deprecated samples argument is gone.
• Posterior predictive check. Simulate new data from the posterior and compare to what was observed. pm.sample_posterior_predictive(dt), which gained sample_vars and freeze_vars in PyMC 6.
• plot_ppc_dist. The ArviZ 1.x plot that lays the predictive draws over the observed data. It replaces the old plot_ppc and, like every ArviZ 1.x plot, returns a PlotCollection.
• Grand rate. 0.141, the chain-wide redemption rate. What complete pooling predicts for every outlet.
• Data weight. n / (n + 116) under partial pooling on the Halloway data: 0.13 at Northgate, 0.89 at Westfield. How far the posterior predictive for an outlet is allowed to leave the centre of 0.135.
• What a check cannot catch. A model wrong in a way the data confirm. Northgate at 0.124 against 0.192, because 1 of 18 is what 0.192 looks like at that size.
• Diagnostics. R-hat, ESS, divergences: proof the sampler explored this model's posterior. Not proof of the model.
• LOO cross-validation. Named, not taught: the tool for comparing models by predictive accuracy on held-out points.

Beta(40, 60) has mean 0.400 and is worth 100 pseudo-observations. Run the Westfield model forwards from it and the counts it produces for 900 vouchers sit around 40 percent redemption. Now the only fact you need: across the twelve outlets, in the month on the table, the highest observed rate anywhere is Hillcrest's 9 of 40, 0.225. The prior is predicting a business that Halloway has never once been, at any outlet, at any size. It has failed, and it failed with no data used, because the check never looked at Westfield's 127. It looked at what the prior expects and asked whether that could be a bakery chain.

This is what makes the prior predictive the shared ground of leader session 5's argument. The subjectivist says the prior encodes belief and nothing else; fine, but a belief that predicts 40 percent redemption is a belief the believer would disown on sight. The reference-prior analyst wants the data to speak; fine, but the flat prior's predictive, every count from 0 to n equally plausible, is what "letting the data speak" looks like before they do, and it is worth seeing. The weakly-informative analyst regularises toward plausible values and checks by prior prediction; this is the check. Nobody has to concede anything about where priors come from to run it, and the Wrong prior fails under all three positions.

The Flat prior, Beta(1, 1), treats every count from 0 to n as equally plausible. Its predictive is a level strip from zero redemptions to all of them. That is not a failure; it is a statement that the prior contributes two pseudo-observations and no opinion, and seeing the strip is how you confirm the prior you wrote is the prior you meant. Weak, Beta(2, 14), predicts a hill peaking inside the observed range with a long right tail that reaches well past it. Informed, Beta(12, 88), predicts a narrow hill entirely inside the range. Both pass the check in the only sense a prior predictive check has: they predict data that could be Halloway's.

Passing says nothing about whether the prior will be right for a given outlet. Session 2 already showed the Informed prior confidently wrong at Northgate, 0.110 with an interval of 0.068 to 0.159 against a truth of 0.192. It passed tonight's check anyway, because Northgate's truth of 0.192 is inside the range Halloway has seen and a prior centred at 0.120 predicts data like Halloway's. A prior predictive check catches a prior that predicts an impossible business. It does not catch a prior that predicts a plausible business and happens to be wrong about one shop. Hold onto that shape of sentence; Part 2 needs it again.

In PyMC 6, pm.sample_prior_predictive() is called bare inside the model block, before pm.sample(). Its deprecated samples argument was removed in 6.0.0, so a 2023 notebook that asked for a particular number of prior draws by that name stops at that line. The observed data are declared in the block so the model knows the shape of what it is predicting, 900 vouchers at Westfield, but the prior predictive does not condition on the 127. That is the whole point: the counts it returns are what the prior alone would produce.

Because it is one call and no sampler runs, there is no reason not to do it every time, and session 5's six-move walk already has it as move two. Leader session 6 tells a commissioning executive to ask whether a prior predictive simulation was run and reported as the range of counts the prior expects. Tonight is the session that produces the thing they are asking for.

Complete pooling has one rate, the grand rate 0.141, and predicts every outlet's fresh month from it. Northgate's observed 1 of 18, 0.056, sits below that; Hillcrest's observed 9 of 40, 0.225, sits well above it. Under a single shared rate both months look like surprises, in opposite directions, and a posterior predictive check on complete pooling shows two outlets the model would not readily have produced. That is the check doing its job: the model's picture of the data is too tidy, and the twelve outlets are not all copies of 0.141.

Partial pooling, session 6's fit with mu = 0.135 and kappa = 116, predicts each outlet from its own shrunk rate. Northgate's is 0.124, its own month weighted at 18 / (18 + 116), so the fresh months it expects are lower and wider than complete pooling's, and one in eighteen sits in a fatter tail. Hillcrest, at n = 40, gets a data weight of 40 / (40 + 116), larger than Northgate's 0.13, so its predictive is allowed further from the centre toward its 0.225 and nine of forty is less of a surprise there too. The check cannot say partial pooling is right. It can say complete pooling has missed a shape in the data and partial pooling has not, and that is the kind of sentence McElreath's lecture 7 builds the check to produce. az.plot_ppc_dist(dt) draws exactly this comparison for any model in the tree.

Session 6's partial pooling passed every check. Its diagnostics were clean, its predictive months at every outlet looked like the months on the table, and its ledger against the truth was the best of the three estimators, 0.025 root-mean-square error. It still put Northgate at 0.124 against a true rate of 0.192, and no predictive check could have told you so. Northgate's one redemption in eighteen is what a true 0.192 looks like at that size, so a model that predicts fresh months centred near 0.124 produces months that look like the observed one. The observed data confirm the model. The model is wrong about the one shop that mattered.

This is the sentence to carry out of tonight, and it is the honest limit of the whole idea. A predictive check compares the model to the data you have, and the data you have were small. It catches a model that could not have produced these data: the Wrong prior at Westfield, complete pooling at Northgate and Hillcrest. It cannot catch a model that produces these data comfortably and is wrong anyway, because the only evidence against it is a truth the data are too few to show. The same goes for the diagnostics, in the other direction: a passing R-hat and ESS set proves the sampler explored this model's posterior, and nothing about whether the model is right, which is exactly what tonight's checks are for. Two layers, two limits, and neither one is the truth.

Three things tonight names and leaves. LOO cross-validation is the tool for comparing two or more models by their predictive accuracy on held-out points; when the question is "which of these models predicts better", that is the answer, and it is a subject of its own. Bayes factors and comparison by marginal likelihood are a different way of comparing models, and they are what Kruschke's ROPE rule in session 7 is explicitly not: that rule is estimation against a band of practical equivalence, not a ratio of model evidences. And McElreath's lectures 14 to 18 are the road onward from twelve Beta-Binomial outlets: Gaussian processes, measurement error, missing data, social networks. Each is the same machine, a model run forwards to produce data, pointed at a harder shape of data.

One thing tonight does not do at all. Whether the 89 percent intervals these models report are honest over many forecasts, measured against what happened, is a calibration question, and learn-decision-intelligence-with-phoebe sessions 02 and 04 own it. A posterior predictive check asks whether the model could have made this month's data. Calibration asks whether its stated probabilities come true at the stated rate across many months. They are different questions with different owners.