Where session 1 left you
Your memo has a default and a flip-condition: "launch IF expected net gain exceeds $4k under our own estimates." The phrase doing the load-bearing is our own estimates - and an estimate from an uncalibrated estimator is a confident-sounding unknown. Tonight instruments the estimator: you.
★ What you walk out with tonight. Your own measured calibration number - the single most useful uncomfortable fact this course produces. The half-Brier scoring rule that makes "good forecast" a computable property. And the width-is-honesty discipline that turns estimates into things a memo can actually use.
LiveWhat calibration means, precisely3 min▶
You are calibrated when your stated confidence matches your long-run hit rate: of all the things you call "90 percent sure", about 90 percent turn out true. Not smarter, not more certain - matched. A calibrated person's "60 percent" and "95 percent" are different, reliable products; an uncalibrated person's numbers are mood wearing digits.
The property is checkable, which is what makes it a skill rather than a personality trait. Tonight's lab checks it the standard way: many interval estimates, verified answers, count the hits.
Mango Lane's ops lead estimated the checkout revamp at "six weeks, definitely" - it took eleven. The same lead's "risky, maybe 50-50" migration shipped clean. Neither surprised anyone who had counted: the lead's confident estimates run late about half the time. That count is calibration measurement, done informally - tonight just does it honestly, on you, with receipts.
The overconfidence result, with its receipts
7 min liveThe claim "people are overconfident" is folk wisdom; the measured version has a shape, a size, and a fix. Three layers of evidence, each characterized honestly - because a course about calibrated claims has to calibrate its own.
LiveThe three layers of evidence, each labeled honestly5 min▶
- The peer-reviewed anchor. Lichtenstein, Fischhoff and Phillips' 1982 review "Calibration of probabilities: The state of the art to 1980" - the classic synthesis in the Kahneman-Slovic-Tversky volume - documented pervasive overconfidence in interval estimates across studies and populations. This is the load-bearing citation.
- The practitioner numbers. Douglas Hubbard's How to Measure Anything reports untrained subjects capturing the truth in some studies only about half the time on claimed-90% intervals, approaching 90 after calibration practice; his training deck states that "a small amount of training (three hours) can significantly improve calibration of estimates". Honest label: these are his workshop data plus his reading of the literature, not a peer-reviewed evaluation of his method - useful, vivid, and characterized as exactly what they are.
- The tournament-grade result. In the largest forecasting tournament ever run, a probabilistic-reasoning module - about 45 minutes, teaching reference classes and bias correction - measurably improved forecasters' accuracy, with benefits lasting across periods of 8 to 10 months. Session 3 gives that study its full treatment.
Why the range 40-60 and never a single number. Untrained hit rates vary by study and population. A course that said "exactly 50 percent" would be committing the very sin it teaches against: false precision where the honest statement is an interval. The claim's own uncertainty, stated - practice what session 6 preaches.
Self-studyThe half-Brier score: making "good forecast" computable4 min▶
Glenn Brier's 1950 rule scores a probability forecast against what happened: squared error between the stated probability and the 0-or-1 outcome. This course uses the modern half-Brier convention throughout - one squared term, range 0 to 1 - rather than Brier's original category-summed version, whose binary range is 0 to 2. Same idea, half the number; worth knowing both exist because the tournament literature uses the original.
| You said | It happened? | Half-Brier | Reading |
|---|---|---|---|
| 90% | yes | (0.9 - 1)^2 = 0.01 | confident and right: nearly free |
| 90% | no | (0.9 - 0)^2 = 0.81 | confident and wrong: brutal |
| 50% | either | 0.25 | the coin-flip floor - shrug quantified |
| 60% | yes | 0.16 | mild lean, mildly rewarded |
Two properties make it the standard. It is proper: your expected score is best when you report what you truly believe - hedging and bravado both cost. And its asymmetry is the calibration lesson in one row: being 90-and-wrong costs 81 times being 90-and-right. Confidence is a loan against reality, at interest.
The half-Brier scale, at a glance
4 min liveBefore the lab scores you, know the ruler. One picture holds the whole scoring rule and its moral.
The lab: your number, measured live
★ 22 min · everyone buildsTen questions below, spanning mountains to physics. For each, enter a range you are 90 percent sure contains the true answer. The discipline that matters: answer ALL ten, including the ones you know nothing about - cluelessness is not an excuse, it is a signal to widen. Then press score.
Answer all ten before scoring - no lookups, no skipping the scary ones. A skipped question is the estimator refusing to be measured, which is exactly the behavior that keeps organizations uncalibrated. Wide honest ranges on unfamiliar questions are correct technique, not weakness.
Press score and sit with your number for ten seconds before rationalizing. If you claimed 90 and hit 4 or 5 of 10, your "90 percent sure" is currently a coin flip wearing a suit - and you are in the statistical company of nearly everyone untrained. The number is a baseline, not a verdict.
Read your misses' pattern. Most first-timers miss the same way: ranges anchored on a half-remembered number, then narrowed to sound knowledgeable. The reveal links show each true value's source - including one whose "true value" carries its own error bar, which is tonight's bonus lesson about the world.
Apply the equivalent-bet test to your three narrowest ranges. For each: would you genuinely bet $9 to win $1 that the truth is inside? That is what 90 percent means. If your stomach says no, your range was performance, not probability - widen until the bet feels fair.
Now write the memo's section 3, with technique. The voucher decision needs two estimates: campaign uptake rate and per-customer retained value. Write each as a 90 percent interval using tonight's discipline - anchored wide, narrowed only by actual knowledge. Label each "from an estimator measured at N percent tonight" - the memo carries your calibration number as metadata, which is what honest estimates look like.
The retake rule. Calibration training is feedback plus repetition: retake the lab in a week (the answers will have faded), and track your number across attempts. Hubbard's workshops report most people approaching 90 within hours of this exact loop. The lab stays here; the skill compounds.
The width of an interval is a statement of knowledge
6 min liveThe lab exposes the core confusion overconfidence feeds on: treating narrow ranges as competence. The truth runs the other way - the width IS the information.
LiveReading intervals in the wild, starting tomorrow4 min▶
Once you hold the width-is-honesty frame, meetings change. Three habits worth installing:
- Ask for the interval, not the number. "Six weeks" becomes "four to eleven weeks, 90 percent" - and suddenly the plan has contingency where the false precision used to be. A point estimate is an interval with its honesty amputated.
- Track hit rates, not vibes. The colleague whose "definitely" runs 50-50 and the quiet one whose ranges always contain the truth - only counting reveals which is which. A simple log of stated intervals versus outcomes is a team calibration lab, free.
- Reward the wide-and-right over the narrow-and-wrong. Most orgs do the opposite: confidence gets the promotion, honesty gets "why so vague". The Brier asymmetry - 0.81 versus 0.01 - is the argument to bring: confident-and-wrong is not near-right, it is the worst score on the board.
The Challenger Deep question in tonight's lab hides the deepest lesson: the 2021 survey's own answer carries a plus-minus 6 meter error bar, and other modern surveys differ by tens of meters. Even "the true value" is an interval when you look closely enough. An estimator uncomfortable with width is uncomfortable with how knowledge actually works.
Self-studyWhy the lab works: the feedback loop that trains calibration3 min▶
Calibration training is unusually reliable for a cognitive skill, and the mechanism is mundane: scored repetition with immediate feedback. Everyday life never provides it - your confident claims are rarely checked, and the checks arrive too late to attribute. The lab compresses the loop to seconds.
- Repetition: many estimates, so the hit rate is a real frequency rather than an anecdote.
- Verified feedback: every reveal carries its source - no arguing with the answer key.
- The equivalent bet: converting "90 percent sure" into "bet $9 to win $1" recruits the part of your judgment that handles stakes honestly - Hubbard's most portable trick.
- Spaced retakes: the training literature's durability finding - effects lasting months from under an hour of practice - is the reason the retake rule exists.
Before session 3
30 min- Retake the lab in three days. Log both numbers. One retake is worth more than this entire page reread twice.
- Make ten work estimates as 90 percent intervals - ticket counts, delivery dates, metric values you will learn next week. Score them as answers arrive. This is your personal calibration log, started.
- Run the equivalent-bet test in one real meeting. When someone says "90 percent sure", ask - kindly - whether they would take the $9-to-win-$1 bet. Watch the range widen in real time.
- Compute three half-Brier scores by hand from your own week: one confident-right, one confident-wrong, one honest 50. Feel the 0.01 / 0.81 / 0.25 asymmetry once with your own events.
- Optional: read Hubbard's chapter 5 on calibrated estimates - the equivalent-bet test and the pro-con technique in full.
Three questions
Clear all three and session 2 gets stamped in your passport.
1 · What does the calibration literature actually say about untrained people's "90 percent confidence intervals"?
The range matters twice: it is what the evidence says, and stating a range instead of a false constant is the course practicing its own discipline. Hubbard's ~50% figure is real but workshop-sourced; the academic anchor is the 1982 review documenting pervasive interval overconfidence.
2 · You forecast 90 percent and the event does not happen. Under the half-Brier score this course uses, what did that forecast cost versus being 90-and-right?
(0.9 - 0)^2 = 0.81 against (0.9 - 1)^2 = 0.01. The squaring is the point: it makes the rule proper (truthful reporting is optimal) and prices bravado honestly. Note the convention: this is the modern half-Brier (0 to 1); Brier's 1950 original sums over categories and runs 0 to 2 for binary events.
3 · Two analysts estimate campaign uptake. A says "11 to 13 percent" and has missed 5 of her last 10 claimed-90 intervals. B says "6 to 20 percent" and has hit 9 of 10. Whose estimate should the memo carry?
An interval's value is its width TIMES its reliability, and only the hit rate certifies the reliability. A's range sounds better in the meeting and is measurably a coin flip. This is also why the memo records the estimator's calibration number as metadata - and why naive averaging with an uncalibrated input imports the miscalibration.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 2 cheat sheet · pin this
You can now state honest uncertainty about a quantity. Session 3 turns to events: the forecasting habits that won the largest tournament ever run - base rates before case details, small frequent updates, the fox's many small ideas - plus the famous dart-throwing chimpanzee quote taught with the correction its own author begs people to remember. Then the forecast game scores you against two bots, and one of them is humblingly hard to beat.