The most expensive sentence in AI product review
"The bot is about ninety percent accurate." Everyone nods. The number is not wrong, nobody is exaggerating, and the launch that follows still surprises everyone in month two. The gap is not honesty and it is not skill - it is that the speaker measured something and the room heard a guarantee, and no one in the conversation owned the difference. That difference has a size, it is computable, and by the end of this session you will be able to compute it in your head.
One number, three claims, and only one of them is evidence 7 min live
When someone says a bot is 90% accurate, ask yourself which of these three they mean. In almost every meeting the speaker means the first, the listener hears the second, and the decision is actually made on the third.
LiveWhy the lower bound, and not the interval, is what you say out loud3 min▶
A launch decision is one-sided. You do not care whether the bot might secretly be better than measured; you care whether it might be worse than the bar. So the useful half of the interval is the bottom of it, and the sentence that carries a decision is always shaped like "at least X, with 95% confidence".
This also fixes the most common review-meeting failure, where a team reports a range and the room quietly rounds it to the middle. Give people a floor and they argue about the floor, which is the right argument. Give them a range and they argue about the midpoint, which is the number nobody should be using.
One honest caveat to carry: the interval covers sampling error only. It says what your set of cases can prove about the population your set was drawn from. It says nothing about a set drawn from the wrong population, a judge that grades generously, or a distribution that shifts next month. A beautifully narrow interval around a badly sampled set is a precise answer to the wrong question - which is exactly why session a3 comes before this one.
Self-studyWilson, not Wald: why the arithmetic on this page uses one formula2 min read▶
There is more than one way to put an interval around a proportion. The textbook one most people half-remember (the Wald interval) behaves badly at exactly the values that matter here: small samples and rates near 90% or 100%. It can produce a bound above 100%, and at 20 for 20 it produces an interval of zero width, which would tell you a perfect score on twenty cases proves perfection.
The Wilson score interval is the standard recommendation for this situation, set out in the reference paper by Brown, Cai and DasGupta in Statistical Science (2001). It stays inside 0 and 1, behaves sensibly at the extremes, and is what every figure on this page and in this course's simulator is computed with. You do not need the formula. You need to know that one formula was used consistently, and which one, so that your numbers and your team's numbers can be compared.
The width of the bar, and the price of narrowing it 6 min live
Hold the observed rate fixed at 90% and change only the number of cases. Nothing about the bot varies in this picture. Everything that moves is a statement about how much you looked.
Read it as a ladder of permissions rather than a table of statistics. Each row is a sentence you are allowed to say:
| Golden set | 95% interval at an observed 90% | What you are allowed to say |
|---|---|---|
| n = 20 | 69.9% - 97.2%, 27.3 points wide | Almost nothing. 18 of 20 is fully compatible with a bot that is truly right seven times in ten. |
| n = 50 | 78.6% - 95.7%, 17.0 points wide | You cannot rule out that it is below 80%. |
| n = 100 | 82.6% - 94.5%, 11.9 points wide | The first defensible "high 80s or better". |
| n = 200 | 85.1% - 93.4%, 8.4 points wide | You can defend a claim of 85% or above. |
| n = 500 | 87.1% - 92.3%, 5.3 points wide | Roughly 4-point regressions become detectable run to run. |
| n = 1000 | 88.0% - 91.7%, 3.7 points wide | "About 90%, give or take two." A T3 bar of 90% is now arguable. |
| n = 2000 | 88.6% - 91.2%, 2.6 points wide | Diminishing returns. Four times the work since n=500, under three points bought. |
The same fact from the other direction, which is the form you will want when someone asks how many cases to label. To achieve a given margin of error around a rate near 90%:
| Margin you want | Cases needed | What that costs in practice |
|---|---|---|
| ±10% | n ≈ 35 | An afternoon. Useful for iteration, never for a launch claim. |
| ±5% | n ≈ 138 | A couple of days. The realistic floor for a first honest report. |
| ±3% | n ≈ 384 | About a week of labelling. Where most serious launch evidence sits. |
| ±2% | n ≈ 864 | Weeks, or a labelling budget. Justified for high-tier bots. |
| ±1% | n ≈ 3,457 | A project of its own. Ask hard whether the decision needs this precision. |
Self-study"Start with 20-50 tasks" is excellent advice about a different activity3 min read▶
Anthropic's engineering guidance on agent evals recommends starting with 20 to 50 simple tasks drawn from real failures. That advice is correct, and the table above does not contradict it - the two are about different jobs, and conflating them is how a 20-case set ends up in a go/no-go deck:
- Iteration. Early changes produce 30-point swings. A 30-point swing is visible at n=30, so a small set drawn from things that actually broke is the fastest possible feedback loop. Starting big here is a waste of a week.
- Launch evidence. You are now claiming a few points of difference from a threshold. That needs an order of magnitude more cases, for the reason the picture above shows: at n=20 the floor sits 20 points below the estimate.
The same guidance draws a second line worth carrying as a leader: capability evals start low and exist to drive improvement, so a low score there is the system working. Regression evals are expected to run at a nearly 100% pass rate, so anything else is a bug rather than a score. If a report merges the two, you will misread both. Builder session b1 splits them properly.
Your golden set has structure, and the arithmetic does not know 7 min live
This is the best gotcha in the course, and it catches sophisticated teams rather than careless ones - you have to have got everything else right before it can bite you. Anthropic's error-bar research puts it plainly: clustered standard errors on popular evals can be over three times as large as naive standard errors. Read that as: the interval you were proud of may be up to three times too narrow.
LiveThe companion recommendation: compare versions with paired differences3 min▶
The same Anthropic research offers the counterpart move, and it is the one that saves you money rather than costing it. When you are comparing two versions of a bot rather than measuring one in isolation, do not compare two independent scores. Run both versions on the same questions and difference them question by question.
The reason it works: a hard question is hard for both versions, and an easy one is easy for both. Differencing per question cancels that shared difficulty out, so you are left looking only at the change you care about. In practice this means you can detect a genuine improvement of a given size with substantially fewer cases than an unpaired comparison would need - the same evidence, better arranged.
As a leader, the two questions this gives you are short. "Were both versions run on the same set?" and "Does your set have repeated units in it - several questions per document, per table, per customer?" Almost nobody expects a leader to ask the second one, and almost every real golden set answers yes.
The live interval table ★ 10 min · read it together
Below is the same table as Part 2, computed live rather than typed out. The observed accuracy is pinned at 90% and only n changes. Read the first row aloud in the session - it is the row that changes how people talk for the rest of the quarter.
Start at n = 20. The pass rate is 90% and the lower bound is 69.9%. Say the consequence out loud: eighteen of twenty is genuinely compatible with a bot that is right about seven times in ten. This is the row that describes almost every demo you have ever been shown.
Walk down to n = 100. The floor has climbed to 82.6% and the width has more than halved. This is the first row where "high 80s or better" is a defensible sentence rather than an optimistic one - and it is also roughly where most teams stop, because a hundred feels like a lot of work.
Compare n = 500 and n = 2000. Four times the labelling, and the width goes from 5.3 points to 2.6. Ask the group the real question: if your bot is one point short of its bar, is it cheaper to improve the bot or to label 1,500 more cases? Teams reflexively pick the first, and it is frequently the more expensive answer.
Now find the row your own most recent eval was on. Most people in the room will be between rows one and three. Write down the sentence that row entitles you to say, and notice how different it is from the sentence that was actually said.
Rewrite a claim you have already made ★ 8 min · at your table
Take the most recent accuracy claim from your own organisation - a slide, a board update, a vendor email you forwarded approvingly - and put it through the four rewrites below. Do it on paper. The point is not to embarrass anyone; it is to feel how much of the original sentence was structure and how much was hope.
1. Name the claim. Is the number in your sentence an observation, a statistical floor, or a feeling? Write which. 2. Attach n. How many cases produced it? If you do not know, that is the finding, and it comes before any argument about the value. 3. Replace the point with the floor. Look up the row in the table and rewrite the sentence as "at least X%, with 95% confidence, on n cases". 4. Check for clusters. Ask whether the set was built as several questions per document, table, ticket or conversation. If it was, say so and widen your claim, because the printed interval assumed something about your set that is not true.
What this sounds like when it lands well. The before: "our support bot is about 90% accurate and we think it is ready." The after: "on 500 cases sampled from December traffic, including a 15% adversarial and unanswerable slice, graded by rule and execution where possible with the judge calibrated against 120 human labels, it passed 474. True task success is at least 92.5% with 95% confidence, against a tier bar of 90%. The set is drawn from 500 separate conversations rather than several questions per conversation, so the interval is not clustered." It is longer. It also ends the meeting, because there is no follow-up question left in it - and that is what you are buying with the extra clause. Builder session b9 assembles this sentence from a live scorecard.
Try it yourself - this week ◐ 45 min total
- Ask for the n behind the last three AI performance numbers your organisation reported. Not the methodology, not a review - just the three counts. The distribution of answers tells you where your measurement culture actually is.
- Take your highest-stakes bot and work backwards from the decision: what margin of error would actually change your mind, and what does the sample-size table say that costs? Bring that number to the team as a budget question rather than a quality complaint.
- Ask the cluster question once, gently, of whichever team you trust most: "is the set several questions per document, or one question per document?" Whatever the answer, you will learn something, and it is a question they will remember being asked.
- Rewrite one claim you personally own into the "at least X%, with 95% confidence, on n cases" form and use it in your next update. Then watch what happens to the questions you get back. Fewer, and better.
- Sit in on one version-comparison review and ask whether both versions ran on the same questions. If not, the team is paying for cases it does not need.
Official sources covered
Every figure on this page traces to a primary source catalogued in this course's source map, dated 2026-08-13. The interval figures are computed rather than quoted, and you can reproduce every one of them. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A team reports that 18 of 20 cases passed, so the bot is 90% accurate. What are you entitled to conclude?
The 90% is a true statement about the twenty cases you ran and almost nothing about the bot. At n=20 the interval is 27.3 points wide, and its floor sits more than twenty points below the headline. This is the arithmetic behind the rule that a demo is not evidence: not because demos are dishonest, but because twenty observations cannot separate a 70% bot from a 95% one. Option C is the instinct to eyeball a margin, and the point of the table is that the real margin is much bigger than instinct suggests.
2 · You are at n=500 with an interval of 87.1% to 92.3%. Someone proposes labelling 1,500 more cases to reach n=2,000. What is the right response?
Interval width shrinks with the square root of n, so buying precision gets steadily more expensive while the bot itself does not improve at all. That does not make the spend wrong - if you are one point short of a tier bar, narrowing the interval is sometimes genuinely the cheapest path to a defensible launch, and builder session b9 shows exactly that case. What makes it a leadership decision is that it is a real trade with two priced options, and it is usually presented as though only one of them existed.
3 · Your team's golden set is 200 questions, built as 10 questions over each of 20 documents. What should you say about the reported interval?
Questions drawn from the same document share their difficulty, ambiguity and retrieval context, so they pass and fail together. The naive formula counts them as 200 independent facts when they behave much more like 20, and Anthropic's error-bar research measures the consequence: clustered standard errors on popular evals can be over three times as large as naive ones. Option C is the trap - doubling to 400 questions over the same 20 documents adds work and adds almost no independent evidence, which is exactly the mistake the finding warns against.