How the leader track works
Six sessions, 45 minutes each, no code. Tonight: what a prior is and what it costs, measured on data whose truth is known. Session 2 is why eighteen observations cannot give you a point and what they can give you instead. Session 3 is borrowing strength, the reason twelve small outlets know more together than any one of them alone. Session 4 is reading a posterior in a meeting: the probability B beats A, and why 0.915 was not enough. Session 5 is the argument about priors that the field has not settled, taught as the argument it is. Session 6 is commissioning a Bayesian answer: what to require in writing so the prior is a decision rather than a default. The example throughout is Halloway Bakeries and its voucher.
★ What you walk out with tonight. One conversion that makes every prior legible: a prior is a number of pseudo-observations, and you can ask for that number in a meeting. The measured result that a prior worth 100 observations still moved an answer by 0.026 at n = 900, which is the end of "the data will wash it out". And the counterweight you need before session 3 makes the case for priors: at Northgate the sensible prior was confidently wrong, and a flat one, for all its vagueness, contained the truth.
LiveA prior is a vote, and it comes with a vote count4 min▶
Every Bayesian estimate blends two things: what you believed before you looked, and what you saw. The blend is not mystical. For a rate, a belief can be written as a Beta distribution with two numbers, a and b, and those two numbers have a plain meaning: the belief behaves exactly as if you had already watched a + b customers, of whom a redeemed. A Beta(12, 88) prior is a hundred imaginary customers, twelve of whom redeemed. That is a hundred votes for a rate near 0.12, cast before a single real customer walked in.
The real data then vote too, one vote per customer, and the posterior mean is the weighted average. Eighteen real votes against a hundred imaginary ones gives the data a weight of 18 / (18 + 100), which is 0.15. The belief carries the other 0.85. Nothing about that is wrong. It is what you asked for when you chose a hundred pseudo-observations, and the only failure is not knowing that you asked for it.
Halloway's analyst set the prior at Beta(12, 88) because the chain-wide rate had been near 0.12 for years and a hundred pseudo-observations "felt about right". Nobody in the approval meeting asked how many observations the belief was worth. Northgate's eighteen real customers were outvoted eighty-five to fifteen, and its posterior landed at 0.110 against a truth of 0.192.
Self-studyThe words that will keep coming back4 min▶
- Prior. What you believed about the rate before this month's data. Written as a distribution, not a number, because you were not certain.
- Pseudo-observations. For a Beta(a, b) prior, a + b. The number of imaginary customers your belief is worth. The single most useful thing to ask about any prior.
- Likelihood. What the data say on their own. At Northgate: one redemption in eighteen.
- Posterior. The blend. Also a distribution, which is why it can be summarised by an interval rather than a point.
- Posterior mean. The weighted average of prior mean and observed rate, weights in proportion to pseudo-observations and real ones.
- 89 percent interval. The range that holds 89 percent of the posterior, cut equally from both tails. It is 89 rather than 95 because that is the default the tooling ships with, for stability, and the number is not sacred.
- Flat prior. Beta(1, 1): two pseudo-observations, one of each. As close to "no belief" as a prior gets, and not nothing.
- Credible interval. The Bayesian interval. It is the sentence most people think a confidence interval is: there is an 89 percent probability the rate is in here. The frequentist version is a different sentence, and learn-statistics-with-phoebe owns it.
Four beliefs, eighteen customers, one truth
10 min liveThe same eighteen Northgate customers, four different beliefs going in. Because Northgate's true rate is written into the data, each posterior can be scored against it rather than argued about. The result is not the one that makes priors look good, and it is on the page because a course that only showed the flattering case would be selling something.
LiveWhy the sensible prior missed, and why that is the point4 min▶
Beta(12, 88) was not a bad prior. The chain-wide rate really is near 0.12, and most Halloway outlets really do sit close to it. The prior encoded a true fact about the population and then applied it to the one outlet that does not fit the population. Northgate's true rate is 0.192, the highest in the chain, and a prior that puts almost no weight above 0.18 will pull any estimate of Northgate down toward the crowd no matter what eighteen customers do.
The lesson is not "use flat priors". The flat prior's interval runs from 0.020 to 0.222, a range so wide it would not support any decision, and session 3 will show that a prior learned from the other eleven outlets does far better than either. The lesson is narrower and more useful: an informative prior is a bet that this case resembles the cases the prior was built from, and the price of losing that bet is a precise-looking interval that does not contain the answer. You cannot avoid the bet. You can know you are making it.
Self-studyThe interval that misses is the dangerous one3 min▶
Look at the widths. The flat prior's interval is 0.202 wide and honest about it. The informed prior's is 0.092 wide, less than half, and it reads like knowledge. In a slide deck the narrow interval wins every time, and it is the one that excludes the truth.
This is the general shape of the risk with strong priors: they do not make you wrong so much as make you confidently wrong, and confidence is what gets a number acted on. The defence is procedural rather than mathematical. Before believing a narrow interval, ask how many pseudo-observations the prior was worth and where they came from. If the answer is "about a hundred, from the chain-wide history", ask whether there is any reason this outlet might not be chain-wide. At Northgate there was: it is the newest site, in a different kind of neighbourhood, and nobody had asked.
Data do not wash out a prior. They outvote it.
7 min liveThe reassurance people reach for is that with enough data the prior stops mattering. That is true in the limit and misleading before it. Westfield is Halloway's flagship with 900 vouchers a month, fifty times Northgate's volume, and the same four priors applied to Westfield's 127 redemptions do not agree.
LiveThe arithmetic of outvoting3 min▶
At Westfield the data weight is 900 / (900 + 100) = 0.90 for either hundred-observation prior. The informed prior's mean of 0.120 sits close to the observed 0.141, so its tenth of a vote moves the answer by 0.002. The wrong prior's mean of 0.400 sits far away, so the same tenth of a vote moves the answer by 0.026. Same weight, different distance, very different damage.
That is the whole rule. A prior's cost is its weight times its distance from the truth. Weight falls as data arrive, but only as n / (n + pseudo-observations), so a hundred-observation prior is still worth a tenth at nine hundred and a twentieth at nineteen hundred. Distance does not fall at all. If the belief was wrong, more data shrink the damage slowly and never to zero on any dataset you will actually have.
Halloway's finance team argued that the flagship's volume made the prior irrelevant, and used the Westfield number to set the chain-wide budget. Under the sensible prior that was nearly true. Had anyone used the 0.400 prior a rival chain's data had suggested, the flagship estimate would have carried a 0.026 error into a budget built on 900 customers a month, and nobody would have seen it, because 900 sounds like enough.
Self-studyThe question to ask about any prior3 min▶
Two questions, in order. How many pseudo-observations is this prior worth, and where did they come from. The first turns a vague word into a number that can be compared to the sample size in front of you. The second turns the number into a bet you can evaluate: if the pseudo-observations come from the chain-wide history and the outlet is unlike the chain, the bet is bad regardless of how carefully it was made.
A prior nobody can express as a pseudo-count has not been chosen; it has been defaulted. Session 5 takes up the argument about how to choose well. Session 3 shows the case where the prior is learned from the other outlets instead of asserted, which is where the discipline earns its reputation. Tonight's job was narrower: to make sure that when session 3 shows a prior working, you already know what it costs when it does not.
Price a prior you are already using
20 min build-alongNo laptop. Pick one small-data number your area acts on, a conversion rate on a new product, a defect rate on a short run, a response rate in a small region, and find the belief that is already sitting behind it. There always is one. The build-along is about making it visible and putting a vote count on it.
Name the rate and the sample. "Redemption at Northgate, eighteen vouchers, one redeemed." If you cannot state the sample size, that is the first finding: somebody is acting on a rate whose evidence nobody has counted.
Find the belief. What did people expect before the number came in, and why. "About 0.12, because that is what the chain does." Write it as a rate and a reason, not as a feeling.
Put a vote count on it. How many customers' worth of evidence is that belief? Ten? A hundred? A thousand? Most people, asked directly, land somewhere between fifty and two hundred. Write the number down before anyone can revise it.
Compute the weight. Real sample over real sample plus pseudo-count. Eighteen over one hundred and eighteen is 0.15. Say the result out loud: "the evidence carries fifteen percent of this estimate".
Ask where the pseudo-observations came from. The chain? Last year? A rival? A slide? Then ask whether this case is like that source. If there is a specific reason it might not be, you have found the Northgate risk.
Write the sentence. "This estimate is [weight] evidence and [1 minus weight] belief, and the belief comes from [source], which may not apply because [reason]." If the sentence is uncomfortable, the estimate was already uncomfortable; you have only made it visible.
The question that changes the meeting. "How many observations is that belief worth?" It has no technical words in it, it can be asked of anyone, and it has never once been answered with a number smaller than the sample in the room without somebody's face changing.
Three questions
An analyst describes their prior as Beta(12, 88). A colleague says that sounds reasonable and moves on. What has the colleague not asked?
Beta(12, 88) behaves as if a hundred customers had already been observed. Against eighteen real ones the belief carries 85 percent of the estimate. That ratio, not the family or the interval level, is what decides whether the prior is doing the talking.
Westfield has 900 vouchers a month. A team argues that at this volume the choice of prior cannot matter. On the measured data, what is true?
The prior's weight at n = 900 is 100 / 1000. What that tenth of a vote costs depends on how far the prior sits from the truth: 0.0021 for the informed prior, 0.0259 for the wrong one. Volume shrinks the weight; it does nothing to the distance.
At Northgate the informed prior produced an interval of 0.068 to 0.159 and the truth was 0.192. What is the right lesson?
The prior encoded a true fact about the chain and applied it to the one outlet unlike the chain. Session 3 shows that a prior learned from the other eleven outlets does much better than either the informed or the flat one; the fix is a better bet, not no bet. Widening to 95 percent would not have reached 0.192 either.
Before session 2
30 minTake the number you priced tonight and find out two things about the sample behind it. How many observations there actually were, and whether anyone has ever stated an interval for it rather than a point. Bring both. Session 2 is about why eighteen observations cannot give you a point at all, and what the honest alternative looks like when it has to go on a slide.