The leader track, and what it is for
Six sessions, no code, no tools. The builder track next door measures bots; this track decides about them. The whole course reduces to one question that lands on your desk and nobody else's: given all these numbers, do you ship? Today is about why that question is hard, and why the answer that feels obvious in a demo room is so often wrong by the following Tuesday.
A demo and a Tuesday are different experiments 7 min live
The failure is almost never that somebody lied. The failure is that a demo answers a question nobody in the room realised was different from the question they were asking. Read the two columns below as two experiments with different subjects, different sample sizes and different observers.
LiveThe three sentences that get confused in a launch meeting4 min▶
Listen for these in your next review. They sound like paraphrases of each other. They are three different claims with three different evidence requirements, and the meeting usually starts with the first and ends by acting on the third.
- "It can do this." A demo proves it. One success is sufficient evidence. This is a real and valuable claim - it is how you decide whether to fund the next quarter.
- "It does this reliably." Now you need many attempts, on questions the builder did not choose, and a rate rather than an anecdote.
- "It does this reliably enough to be safe unattended." Now you need a rate, an honest error bar around the rate, a threshold agreed before anyone measured, and a plan for the cases in the tail.
Nobody moves between these deliberately. The slide says the first, everyone hears the second, and the launch date is set as though the third had been established.
Self-studyWhy multi-turn is the row that catches most teams3 min read▶
Row three of the diagram is the quietest and the most expensive. Demos are single-turn because a single turn is what fits in a slide. Real support conversations are not: the customer clarifies, contradicts themselves, and refers back to something they said four messages ago.
The published gap is large. Across a body of multi-turn benchmark work, models show roughly a 39% average performance drop moving from single-turn to multi-turn versions of the same tasks, and the research attributes it to unreliability rather than lack of ability - the model can do it, and then on some turns it does not. On MultiChallenge, frontier models sit under 50%, with the best result at 41.4%. Session b5 measures this properly on Beacon's support bot; for now, treat any single-turn number you are shown as an upper bound on the number that matters.
Klarna, both halves 6 min live
This is the most cited AI support deployment in the world, and it is normally cited at the halfway point. The first half is real. The second half is the lesson, and the two together are worth more than either alone.
What made act 1 hard to argue with, and why that was the problem. The figures were specific, operational and independently plausible: volume, handle time, repeat-contact rate, a projected profit line. That is a better class of evidence than most launch decks contain. It was still a measurement of a particular month, on a customer mix that a brand-new deployment attracts, judged against a definition of "handled" that the deploying company set. None of those three caveats is a criticism. All three are questions you should be able to ask about your own numbers, out loud, without anybody in the room taking it personally.
What a wrong launch costs, and what counts as evidence 8 min live
Two numbers make this decision worth arguing about. The first is how much cheaper a bot interaction is than a human one, which is why the project exists. The second is what the industry has quietly started charging for, which tells you where the risk actually sits.
| Unit of cost | Figure | How solid is it |
|---|---|---|
| A human-assisted contact, US | ~$13.50 median | Gartner-attributed and reaching us second hand. Directionally quoted everywhere; treat it as an order of magnitude, not a line item. |
| A bot interaction | ~$0.50 - $0.70 | Widely quoted industry range, secondary. The roughly 20x gap to a human contact is the entire business case, and it is real. |
| Intercom Fin, per resolution | $0.99 | The vendor's own published pricing. Primary - and notably, the definition of what counts as a resolution is published alongside it. |
| Sierra, per resolution | ~$1.50 | Secondary. Same shape as Fin: you are billed for outcomes, not for conversations. |
| The structural shift | per interaction → per outcome | The pricing model moved from charging per interaction to charging per outcome. That only works if the outcome can be measured - which is what this course is about. |
Read the last row twice. When a vendor bills per resolution rather than per conversation, they have taken on the measurement risk and they have had to define resolution precisely enough to invoice against it. Your internal bot has neither of those forcing functions. Nobody sends you a bill that makes you define what "handled" means.
LiveWhy "it demoed well" survives so long as an argument3 min▶
Because it is the only evidence in the room that everybody can personally verify. A confidence interval requires trusting somebody's arithmetic; a demo requires only eyes. That asymmetry is why the weakest rung on the ladder is also the most persuasive one, and why the fix is not a better demo but a better question.
There is a second reason, and it is worth naming in front of the team: a demo that fails is embarrassing in a way that a bad eval score is not. So demos get rehearsed, and rehearsal is exactly the thing that destroys their value as evidence. The more polished the demo, the less it tells you.
Self-studyThe cost of being wrong is not symmetric3 min read▶
Launching too late costs you the difference between roughly $13.50 and roughly $0.60 per contact, multiplied by volume, multiplied by weeks. It is a large, legible, spreadsheet-shaped number, and every finance partner can compute it.
Launching too early costs you something that does not fit in the same spreadsheet: a customer who was told the wrong refund policy by a system that sounded certain, a screenshot that outlives the incident, a regulator's question about how the decision was made. Both costs are real. Only one of them has a champion in the room, which is precisely why the launch gate has to be written down in advance rather than debated on the day.
The demo autopsy ★ 9 min · everyone discusses
Take the last bot demo you personally sat through. Not a hypothetical one - a real meeting, with a real date. Work through these five questions in pairs, then compare answers. Most people cannot answer three of them, and that is the finding.
Who chose the questions? Name the person. If it was the person who built the bot, you watched a proof of possibility. That is worth funding a next step and worth nothing as launch evidence.
How many questions were there, and how many turns each? Write both numbers down. If you cannot remember, that is your answer - it was few enough that counting felt unnecessary.
Did anything in the demo fail? A demo with zero failures was either rehearsed or too easy. Both mean the same thing for your purposes. A demo where one thing went wrong and was handled gracefully is worth more than a flawless one, and almost nobody presents it that way.
Was there a single question the bot should have refused? Something outside its knowledge, or a request it is not allowed to fulfil. If not, you have no evidence at all about whether it can say "I do not know" - which is the behaviour that decides whether it is safe.
What decision did that meeting actually make? Compare it to what the evidence supported. If a launch date came out of a proof of possibility, you have found the gap this whole course exists to close - and you have found it in your own calendar, not in a case study.
Price your own wrong launch ★ 7 min · everyone writes
Both sides of the decision have a price. Only one of them usually gets a number. Fill in both columns for a bot in your own organisation, in the next seven minutes, roughly. Precision is not the point; having a number on the right-hand side at all is the point.
Cost of launching six weeks late: contacts per week × (your cost per assisted contact, anchor ~$13.50 in the US) × the share the bot would deflect × 6. Add the competitive or morale cost if it is real, and say so out loud rather than folding it into the arithmetic.
Cost of launching six weeks early: wrong answers per week (volume × your best guess at the error rate) × how many of those are irreversible × what one irreversible wrong answer costs. Then, separately and without a number: what is the worst single sentence this bot could say to a customer, and who would be reading it?
Write the left-hand number first. It is easy, and everybody in your organisation already has it. Notice how quickly the room agrees on it.
Now write the right-hand number. Notice that you had to invent an error rate to do it, because nobody has measured one. That missing error rate is the whole of session a4, and it is why a leader track exists at all.
Finally, name the one irreversible action your bot can take unattended. For Beacon's support bot it is issuing a refund credit. If your bot has one, your launch bar just went up, and session a5 will tell you by how much.
Try it yourself - this week ◐ 35-45 min total
- Ask your team for the exact list of questions used in the last demo. Not a summary - the list. Count them, count the turns, and count how many the bot should have refused. The gap between what you expected and what you receive is the most useful half hour you will spend this month.
- Find the case study that appears in your own business case and go looking for its act 2. Search for the same company plus the word "hybrid", or plus a year that is two years after the announcement. If there is no act 2 yet, note the date of act 1 and how recent it is.
- Fill in the right-hand column of the two-column estimate properly, with the support or risk lead in the room. Bring the number to your next launch review whether or not anybody asked for it.
- Write down, in one sentence, the worst thing your bot can do unattended. Keep the sentence. Session a5 turns it into a risk tier, and the tier sets your launch bar - so this one sentence quietly determines how much evidence you are about to need.
Official sources covered
Every figure on this page traces to a primary source catalogued in this course's source map, and secondary claims are labelled as secondary on the page itself rather than only here. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · What is the lesson of the Klarna arc for your own launch decision?
Both halves are true at once, and that is the point. The first-month numbers were specific, operational and reported by the company that ran the deployment. The later move back toward a human plus AI hybrid does not retract them; it shows what a launch metric is and is not. Ask of any case study, including your own: which conversations were counted, over what window, and against whose definition of success.
2 · Vendors have moved from charging per interaction to charging per resolution or outcome. Why should that change how you think about your internal bot?
A vendor billing $0.99 per resolution has had to publish what a resolution is, down to whether a customer returning to the thread reverses it. Nobody sends your internal team that invoice, so nothing forces the definition to be written down - and an undefined success metric will drift toward whatever makes the dashboard look good. Per-outcome pricing only works when the outcome is measurable, which is the same requirement a launch gate has.
3 · A team shows you a flawless fifteen-minute demo. What is the single most useful thing you can ask?
A flawless demo is either rehearsed or too easy, and both mean the same thing as evidence. The three sub-questions in option A each attack a different column of the gap: who chose the questions attacks representativeness, how many attacks sample size, and did anyone try to break it attacks the missing adversarial slice. Microsoft ran hundreds of expert hours of red teaming on Bing Chat before launch; the honest comparison is between that and a rehearsed quarter-hour.