You are not being lied to. You are being shown a different question's answer
Almost every misleading eval number in this session was produced honestly, by competent people, using a defensible method. That is what makes them dangerous. A leader who assumes bad faith looks for the wrong thing; the actual failure is that a number arrives without the definition, the sample and the grading rule that gave it its shape - and those three things move it further than any model upgrade ever will.
The word moves the number about ten times further than the model does 8 min live
This is the headline of the whole session, and it is the one thing to remember if you remember nothing else. Hallucination rate is the clearest published example because two independent efforts have scored the same models under two different definitions, and the results do not overlap at all.
LiveThe same effect, in every metric you will be shown4 min▶
Hallucination rate is the loudest case, not a special one. Every headline metric in this field hides a definitional dial, and the dial is usually set by whoever needs the number:
- Resolution rate. Intercom publishes its definition openly and it is instructive: a hard resolution needs the customer to confirm, a soft resolution counts an exit plus 24 hours of inactivity, the resolution is reversed if the customer comes back to the thread, and a greeting-only exchange never counts. Three of those four rules are choices, and each one moves the rate. Their production range across customers runs roughly 38% to 72% (a secondary, vendor-published figure).
- Accuracy on a query bot. Exact-string match against a single reference answer, versus running both queries and comparing the returned rows, are different metrics with the same name. The first rejects correct answers phrased differently.
- Task success on an agent. Did one attempt succeed, or did all of them? Those are different numbers with the same label, and the gap between them is where production lives. Builder session b8 measures it.
The leader move is always the same and it is one sentence long: "read me the definition you used." If nobody in the room can, the number is not ready to be discussed yet.
Self-studyWhat honestly good looks like, so you have an anchor3 min read▶
Leaders get anchored on 99% because that is what software is like, and a bot report at 99% should worry you far more than one at 78%. Some published anchors worth carrying in your head, all measured on public benchmarks rather than on your data:
- On BIRD, the main text-to-query benchmark, the best published systems sit around 82% execution accuracy while measured human performance is 92.96%. Humans do not get 100% either.
- On Spider 2.0, built from enterprise-shaped workflows rather than tidy exam questions, a frontier reasoning model scored 17.0% - against 91.2% on the older, simpler Spider 1.0. Same class of model, same task name, a completely different difficulty of set.
- For support chatbots, a mature containment rate of 70-90% is a strong result and a well-configured retrieval chatbot commonly lands at 40-65% (Gartner-attributed, reaching you through third parties rather than from a primary study).
Session a2 covers this ground properly. The point to hold here: when a report shows a number far above these anchors, the most likely explanation is not that your team is exceptional. It is that the set was easier.
Internal benchmarks answer a question you did not ask 7 min live
Vendor accuracy claims are the most-quoted numbers in any buying conversation and the least transferable. The published Snowflake Cortex Analyst result is a useful teaching case precisely because the company documented what it took to get there - which is more transparency than most vendors offer, and enough to see the shape of the problem.
The honest way to use a vendor number. Do not throw it away and do not repeat it. Treat it as a ceiling under ideal conditions and ask for the four things that turn it into a forecast: was the set yours or theirs, what preparation work was assumed, how many attempts per question, and how lenient was the marking. Then ask the one question that settles the buying decision - will you run this on fifty of our own questions, in front of us, first try? Every serious vendor says yes. The answer to that pilot is the only number that belongs in your business case.
Sometimes the ground truth is the thing that is wrong 6 min live
Everything so far assumed that at least the right answers were right. On the two most-cited text-to-query benchmarks in the field, an audit checked that assumption and it did not hold. This is the finding that should change how you hear the phrase "state of the art".
The grilling kit: seven questions, fifteen minutes ★ 12 min · everyone asks
These are the questions, in order, in the words you would actually use. None of them require you to understand the model. Every one of them changes the number if the answer is bad, and a team that can answer all seven has almost certainly done the work.
| Ask this | A good answer sounds like | A worrying answer sounds like |
|---|---|---|
| How many cases were in this run? | "Four hundred and eighty, and here is the sheet." | "A good number." Or any answer under about fifty offered as launch evidence. |
| Where were they sampled from? | "A random sample of December production traffic, in the same category proportions as live." | "We wrote them." Or "the ones we use for demos." |
| What fraction are adversarial or unanswerable? | "About fifteen percent, split between the two, and here is how they scored separately." | "None - we wanted to test what it is meant to do." This is the most common answer and the most revealing. |
| Who or what graded them? | "Rules and execution where possible, a model judge for the rest, humans on the safety cases." | "The model checked its own answers." Or a judge doing work a rule could do. |
| Was the judge calibrated, and against how many human labels? | "Against a hundred and twenty human-labelled cases, and we report where it disagrees." | "We wrote a good prompt for it." An uncalibrated judge reports its own opinion as accuracy. |
| What is the interval around that number? | "The 95% lower bound is 88.1%." Session a4 is entirely about this answer. | "It is ninety percent." A point estimate offered as though it were a floor. |
| Was the threshold set before or after the run? | "Before, in the tier document, dated and signed." | "We felt this was a reasonable bar." Nobody has ever failed a gate they set after seeing the result. |
"Walk me through how a single case was chosen and how it was marked right or wrong - pick any case, I do not mind which." A team with a real eval can answer this in thirty seconds and will enjoy answering it. A team without one will answer with an average.
Spot the picked set ★ 10 min · at your table
Cherry-picking is almost never deliberate. It happens because the cases that are easiest to write down are the ones the team already knows the bot handles, and because a failing case is unpleasant to add to your own test suite. Here are the tells, in the order you will notice them.
The pass rate is suspiciously round and suspiciously high. Real evals produce numbers like 87.3% on sets with awkward sizes. A clean 95% on exactly 20 cases is a demo, and a 100% pass rate means the set is too easy to be informative - the practitioner rule of thumb is that a perfect score is a problem with the eval, not a property of the bot.
There is no unanswerable category. Take the report's category list and look for the row where the correct behaviour is to refuse or escalate. If it is missing, the set cannot detect a bot that never says "I do not know" - and that is the bot that invents a refund policy in week one.
Every category passes at about the same rate. Real bots are lumpy: strong on lookups, weak on multi-turn, terrible on questions that need arithmetic. Uniform performance across genuinely different categories usually means the categories are not genuinely different.
The set has not changed in six months. A golden set that never grows is a set nobody is adding failures to, which means production incidents are not flowing back into it. Ask when the last case was added and what incident it came from.
Now do it for real. Take the most recent eval summary from your own organisation - or a vendor's published claim if you do not have one yet - and run the seven questions and these five tells against it. Write down the two answers you could not get. Those two are your next conversation, and they are more useful than any request for a higher number.
Try it yourself - this week ◐ 40 min total
- Put the seven grilling questions into your standard review template, so they arrive with the report rather than after it. This costs you one edit and changes what your teams prepare.
- Find the last AI accuracy number you personally repeated to someone more senior than you. Go and find its definition. If you cannot find one in fifteen minutes, send a short correction - it will do more for your credibility than the original number did.
- Ask one vendor in your current pipeline for the four scaffolding facts behind their headline accuracy: whose set, what preparation was assumed, how many attempts per question, how lenient the marking. Note how long it takes them to answer.
- Pick the one category your team's golden set is missing - it will usually be unanswerable or adversarial - and ask for it to be added before the next report, not after. You are not asking for a better bot. You are asking for a measurement that could have failed.
Official sources covered
Every figure on this page traces to a primary source catalogued in this course's source map, dated 2026-08-13. Vendor claims are labelled as vendor claims. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Two reports give hallucination rates for the same models: one says 0.7%-4.4%, the other says 7-12%. Which one is wrong?
This is the single most useful fact on the page. The Vectara HHEM leaderboard and the stricter FaithJudge re-scoring evaluate the same models on the same kind of task and land an order of magnitude apart, because they draw the line between "supported" and "unsupported" in different places. The consequence for you is a rule, not a fact: never compare two rates from two reports, and always ask for the definition before you react to the number.
2 · A vendor's published benchmark shows over 90% accuracy on text-to-query. What is the most useful next question?
The documented case shows why. That 90%-plus figure came from an internal 150-question set, with a hand-built semantic model over the schema, agentic retries rather than one attempt, and lenient multi-gold scoring - and on the same set, single-shot GPT-4o scored 51%. The 39-point gap is scaffolding you would have to build and pay for. None of that makes the vendor dishonest; it makes their number an answer to a question about their conditions, not yours. The live pilot is the only number that belongs in a business case.
3 · Your team wants to buy the tool sitting three places higher on a public text-to-query leaderboard. What should you say?
The instrument had not been calibrated. When the answer keys themselves were audited and corrected, re-evaluating sixteen agents moved rankings by up to nine positions - which is more movement than the gap you were about to spend money on. The correct posture is neither belief nor dismissal: a leaderboard result is a claim about a dataset, useful for ruling systems out and useless as a forecast of what happens on your schema. Answer B overcorrects into a different mistake, and a refresh does not fix an unaudited key.