Where session 2 left you
Calibration measured your quantities - how wide your intervals honestly need to be. Tonight is about events: will the campaign clear its bar, will the customer churn, will it happen or not? Same honesty, different muscle - and the memo's base-rates section opens to hold what you build.
★ What you walk out with tonight. The chimp line with its author's own correction attached, so nobody garbles it at you again. Three habits with tournament-grade receipts. And your own Brier score against a bot that only ever says the base rate - the single most clarifying number in forecasting.
LiveThe optimistic headline3 min▶
Before the cautionary tales: the good news is enormous. Forecasting skill is real, identifiable, and trainable. The Good Judgment Project won IARPA's ACE forecasting tournament (2011-2015) decisively - so decisively that rival university teams dropped out after year two. Ordinary volunteers, recruited openly, beat the field on hard geopolitical questions using habits you can learn tonight.
That is the frame for everything that follows. The famous pessimistic finding about experts turns out to be narrower than its headline, and the optimistic finding turns out to be sturdier than it sounds. Both deserve to be quoted exactly.
Self-studyTonight's vocabulary2 min▶
- Outside view - starting from how often things like this happen, before your case's details. The reference class comes first.
- Base rate - the reference class's frequency. The forecast game prints one on every round.
- Granularity - saying 63%, not "likely". A number can be scored; an adjective cannot.
- Resolution - saying different things on different days and being right. Part 3 shows why it is the whole game.
- Half-Brier - session 2's score, 0-1 convention as stated there: squared error of the probability, lower is better. Anatomy: a 0.9 forecast scores 0.01 if right, 0.81 if wrong; a coward's 0.5 always scores 0.25.
The chimp line, taught with its correction
6 min livePhilip Tetlock spent 1984-2004 collecting roughly 82,000 expert predictions on politics and economics, and one sentence from that work escaped into the culture and mutated. Here is the sentence, and here is what its own author says the mutation got wrong.
The line everyone knows: "the average expert was roughly as accurate as a dart-throwing chimpanzee" - and the headline it became: all expert forecasts are useless.
Tetlock's own correction, verbatim: "What my research had shown was that the average expert had done little better than guessing on MANY of the political and economic questions I had posed. 'Many' does not equal all... accuracy fell off the further out experts tried to forecast - approaching the dart-throwing-chimpanzee level three to five years out." And on the headline: "The message became 'all expert forecasts are useless,' which is nonsense."
LiveHedgehogs vs foxes: the part that survived scrutiny4 min▶
The finding under the headline that DID hold up, paraphrasing Tetlock's own framework: cognitive style beat topic expertise. He sorted his experts into two temperaments. Foxes - many small ideas, self-critical, comfortable updating when evidence turns - beat hedgehogs - one big organizing idea, confident, evidence bent to fit. And the sharpest twist: hedgehogs did worst inside their own specialty at long range, where the big idea had the most authority and the least accountability.
Which means the trainable thing is not knowledge - your team already has that. It is the style: hold ideas lightly, update in small steps, and treat your own confidence as a claim to be scored. Everything in Part 2 is that style, operationalized.
Mango Lane's most senior merchandiser predicted "athleisure is over" three years running - one big idea, applied to every category review, never scored. The junior analyst who tracks her own hit rate in a spreadsheet quietly beats her on every seasonal call. Fox beats hedgehog, in-house, every quarter nobody measures it.
Self-studyWhy the misquote matters to your meetings3 min▶
The garbled version licenses two opposite failure modes. Cynics use "experts are chimps" to dismiss every forecast, including the short-range ones that measurably beat chance - so nobody bothers estimating. Boosters hear the correction and swing back to trusting confident experts everywhere, including the long-range calls where average experts genuinely drift to chance. The honest scope kills both:
| Claim | Verdict | What to do in the meeting |
|---|---|---|
| "Forecasting is pointless, experts are chimps" | Wrong - short range beat chance | Demand the forecast, with a number |
| "Our expert is confident, so trust the 5-year vision" | Wrong - that is exactly where chimp-level lives | Ask for the base rate and shorten the horizon |
| "Skill exists and cognitive style predicts it" | What the data showed | Score people, train the style |
The habits that won the tournament
6 min liveThe Good Judgment Project did not just win - it ran experiments on WHY its forecasters improved, published in Mellers et al. 2014. Three drivers, tested against controls, one of them absurdly cheap.
Mellers et al. 2014, verbatim: "Our group tested and found support for three psychological drivers of accuracy: training, teaming, and tracking. Probability training corrected cognitive biases, encouraged forecasters to use reference classes..." And on the cheap one: "although the module took only about 45 min to complete, the benefits lasted across two periods (each about 8 to 10 months long)".
LiveThe five habits, operationalized4 min▶
- Outside view FIRST. Start every forecast at the reference-class rate - "how often do campaigns like ours clear the bar" - before touching the case's details. This is the reference-class move the training module taught, and it is the forecast game's whole strategy.
- Granularity. Superforecasters say 63%, not "likely". A 63 can be scored, compared, and updated by three points; "likely" absorbs any outcome without embarrassment.
- Small, frequent updates. Foxes move 5 points on new evidence, not 40. Big swings mean the previous number was never believed.
- Actively open-minded thinking. Hunt for the strongest case AGAINST your current number, not more support for it - session 1's six-word question, turned inward.
- Teams. The teaming driver: forecasters sharing evidence and challenging each other's numbers beat solo forecasters. Your memo review IS this, if the reviewers argue with the base rate.
Precision note, because this course keeps its own receipts: the ~10% Brier improvement figure attached to that training module comes from Superforecasting and Mellers et al. 2015, NOT the 2014 paper - and the module is about 45 minutes, never "an hour". Safe phrasing: a ~45-minute module measurably improved accuracy for the whole year, roughly 10% on Brier in GJP's reporting.
LiveThe 30% claim: a misquote autopsy3 min▶
You have heard it: "superforecasters beat intelligence analysts with classified data by about 30%". Trace it and it thins out fast: the figure comes from one Washington Post account (Ignatius, 2013) of a leaked, unpublished comparison. It was never a published result. When re-analysts matched the aggregation methods on both sides, the gap shrank to roughly 10% and was not statistically significant.
The honest version is still remarkable: superforecasters at least matched analysts with classified access - open-source volunteers with good habits, holding their own against professionals with secrets. Teach that version. Retire the 30.
Why we autopsy misquotes instead of skipping them. The 30% claim and the chimp line fail the same test tonight's habits install: ask for the source, ask for the scope, prefer the boring precise number to the thrilling round one. Practicing on famous claims is how the reflex forms before your own dashboards need it.
Murphy's decomposition: what your Brier is made of
5 min liveSession 2 introduced the half-Brier score (0-1 convention, as stated there). Murphy's 1973 decomposition splits it into three parts - and once you see the parts, you know exactly what the forecast game is testing.
LiveWhy the base-rate bot embarrasses vibes-forecasters3 min▶
Read the decomposition through the bot's eyes. Reliability is calibration - session 2's skill - and the bot's is perfect: it claims the base rate and, across rounds like these, base rates are what happen. Resolution - saying different things on different days and being right - is where daring pays, and it carries a minus sign: good resolution LOWERS your score. The bot has exactly zero of it. Uncertainty is the base-rate variance of the world itself - nobody escapes it, so it cancels out of every comparison.
So the bot's score is the floor that pure honesty buys with zero insight. To get UNDER it, your forecasts must vary across rounds AND be right about which way to vary - which requires genuinely informative case evidence, correctly read. Vibes vary plenty and are right at random: they buy no resolution and pay extra reliability. That is why the bot beats confident people all night, and why beating IT is the real diploma.
The forecast game: you vs two bots
★ 20 min · everyone buildsTwelve rounds, live. The scenarios are synthetic with seeded outcomes - the widget says so on its face - but every Brier score is real arithmetic on your real forecasts. Each round prints its base rate; four rounds carry cards with case evidence actually worth updating on. The bots are playing the same board.
Play all twelve rounds. Anchor every forecast on the STATED base rate, and nudge only where a card carries case evidence - small moves, fox-sized, 5 to 15 points. Rounds with no real evidence: the discipline is to sit at the rate and feel the itch to be interesting. Do not scratch it.
Read the final board. Coin-flip bot: 0.250, the price of maximal cowardice. Base-rate bot: 0.171, the floor pure honesty buys. Your number is the verdict: under 0.171 means your case-evidence nudges carried real information; over it means the nudges were noise wearing a narrative.
Audit your nudges. Note which hint-cards you moved on and whether each move paid. The pattern to catch: moving on vivid cards that carried no information, and under-moving on dull cards that did. Foxes are made in this audit, not in the forecasting.
Fill the memo's base-rates section for the voucher decision. Write the reference class ("retention campaigns like ours"), its rate, and the ONE case fact that justifies a nudge off it - with the direction and size of the nudge. That section is now the outside view, standing guard over section 3's estimates.
Write your granularity rule. One line in the memo: forecasts are numbers, never "likely". A 63 is checkable and improvable by three points; "likely" survives any outcome unscored - which is exactly why comfortable teams prefer it. Superforecasters do not, and as of tonight neither do you.
Before session 4
30 min- Make three real forecasts as numbers - live questions from your own week, each written as "base rate X%, my forecast Y%, because Z". Resolution dates attached. These join your calibration log from session 2.
- Catch one vibes-forecast in the wild - a "likely" or "almost certainly" in a meeting or memo. Rewrite it privately as a number with a base rate, and note how much harder the honest version was to write.
- Replay the forecast game once with a pure strategy: base rate everywhere, no nudges at all. Watch yourself land at the bot's 0.171 - then decide which nudges you would still defend on the evidence.
- Correct one garbled claim gently - if the chimp line or the 30% figure comes up this week, offer the honest version with its source. Practice for the memo's toughest audience: people who already know the headline.
- Optional deep read: Superforecasting ch. 1 (the correction lives there, in Tetlock's own words) and Mellers et al. 2014 - the full text is open access.
Three questions
Clear all three and session 3 gets stamped in your passport.
1 · A colleague says "Tetlock proved experts are no better than dart-throwing chimps." What is the accurate scope of that finding?
His correction is verbatim canon: little better than guessing on MANY questions, accuracy falling toward chimp level three to five years out - and "The message became 'all expert forecasts are useless,' which is nonsense." Teaching the scope IS the lesson: 1984-2004, ~82k predictions, average experts, long range.
2 · The famous claim that superforecasters beat intelligence analysts with classified data by ~30%: what is its honest status?
Trace the citation and it ends at one 2013 newspaper column, not a paper. The defensible - and still remarkable - claim: open-source volunteers with trained habits held their own against professionals with secrets. Retire the 30; keep the receipt-checking reflex that killed it.
3 · In the forecast game, why is the base-rate bot (0.171) so hard to beat?
Brier = Reliability - Resolution + Uncertainty. The bot pays nothing on reliability and earns nothing on resolution; uncertainty hits everyone equally. Vibes-forecasters vary a lot and are right at random - no resolution earned, extra reliability paid - which is why the dull bot embarrasses them, and why beating it is the real diploma.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 3 cheat sheet · pin this
Your intervals are calibrated and your forecasts are anchored - but they are still promises made on character. Session 4 upgrades them to promises made on mathematics: conformal prediction, the method that wraps ANY model's outputs in intervals guaranteed to contain the truth at a rate you choose, with no distributional assumptions - plus the honest fine print about what kind of guarantee that is and when it breaks. Uncertainty you can promise, in writing.