Where you are in the leader track
Sessions a1 and a2 covered the classic side: why one accuracy number lies, and how the threshold turns precision and recall into money. Tonight the same literacy moves to LLM systems - the chat assistants, RAG search tools, and copilots your teams are shipping now. The metrics have new names, but the leader's job is unchanged: know which question each number answers, know which mistakes it hides, and know what to demand before trusting it. No formulas to memorize - four metrics, one distinction, and one rule about judges.
The quality report, decoded metric by metric 15 min live
Every LLM quality dashboard is built from a handful of recurring metrics. Here is each one as the question it actually answers - plus the trap that catches executives who read the number without the question.
| Metric | Plain-English question | Scale | The trap |
|---|---|---|---|
| Faithfulness / groundedness | "Is every claim in the answer supported by the source material?" | RAGAS: claims supported divided by total claims, 0-1. Microsoft Foundry: judged 1-5, pass at 3. | Same concept, two different scales. A RAGAS 0.83 and a Foundry 4.2 are not comparable - never compare across vendors. |
| Relevance | "Does it answer the question that was asked?" | Usually judged, 1-5 or 0-1. | On-topic is NOT correct. An answer can be perfectly relevant and completely wrong - relevance never checks the facts. |
| Completeness | "Did the answer include everything critical, or did it miss something?" | Judged. Foundry's own framing: groundedness is the precision aspect, completeness is the recall aspect. | The forgotten row. Teams tune faithfulness up by answering less - completeness is the counterweight, and it is often not on the report at all. |
| Hallucination rate | "How often does it make things up?" | Benchmark-dependent - leaderboard leaders sit around 2-3% on HHEM-style tests. | There are TWO different definitions of "making things up," and vendors rarely say which one they counted. Part of tonight's whole point - see below. |
Now the distinction that pays for the whole session. "Hallucination" is one word covering two different failures, and they need two different fixes. World-lies (what TruthfulQA-style benchmarks measure): the model states something false about reality - its parametric memory invented or repeated a falsehood. Document-lies (what HHEM and every faithfulness metric measure): the model says something the provided source material never says - even if it happens to be true in the real world. A leader who conflates them funds the wrong fix.
LiveWorld-lies vs document-lies - the fix depends on the failure6 min▶
Run the two failures through a concrete example. Your RAG assistant answers a policy question:
- Document-lie: the answer says "the refund window is 60 days" - true at a competitor, false in YOUR policy document, which says 30. Faithfulness catches this. The fix is retrieval and grounding: make sure the right passage is fetched and the model is held to it. Buying a bigger model changes little.
- World-lie: with no document in play, the assistant asserts a false general fact - the kind TruthfulQA's adversarial questions are built to trigger. No retrieval fix exists, because there is no document to be faithful to. The fix lives in model choice and training.
- The subtle case: an answer that is TRUE in the real world but appears nowhere in the retrieved source is still a faithfulness failure - the system stopped being grounded, and next time the unsupported claim may not be true.
The meeting question this gives you: when someone reports "hallucination rate," ask "measured against the world, or against the document?" The answer tells you which team - model selection or retrieval - owns the fix.
Self-studyWhy vendor numbers never compare3 min read▶
RAGAS defines faithfulness as an arithmetic ratio: claims in the response supported by the retrieved context, divided by total claims, on a 0-1 scale. Microsoft's Foundry defines groundedness as an LLM-judged 1-5 rating, thresholded to pass/fail at 3. Same underlying concept - is the answer supported by the source? - but different graders, different scales, different pass bars.
- A "0.83" and a "4.2" are not the same quantity in different units - they are different measurements. There is no conversion.
- Metric design is a CHOICE the vendor made. When two tools disagree about the same system, neither is broken - they asked slightly different questions.
- The leader's defense: compare a system against ITSELF over time on ONE fixed metric, not against another vendor's number.
The practitioner track (b6) works both formulas hands-on; your job is only to refuse cross-vendor comparisons in review meetings.
Self-studyThe wider metric family - rows you may also meet3 min read▶
Beyond the big four, RAG dashboards often carry the rest of the RAGAS family. Same method as tonight - translate each into its question:
- Response relevancy - "does the answer address the question asked?" - the relevance row you already know.
- Context precision / context recall - the same two questions from session a2, asked of the RETRIEVAL step: of what was fetched, how much was useful, and of what was needed, how much was fetched. A bad answer often traces back here.
- Noise sensitivity - "does irrelevant fetched material drag the answer off course?"
- Factual correctness - "does the answer match a ground-truth reference?" Note this is measured against a REFERENCE answer, not against the retrieved context - distinct from faithfulness, and the two are routinely conflated in reports.
You do not need to memorize these. You need the reflex: any unfamiliar metric row gets asked "what question do you answer, on what scale, judged by whom?"
The judge behind the score 6 min live
Here is the fact that reframes every number from Part 1: most LLM quality scores are produced by another LLM. A "judge" model reads the answer and grades it - it is the only way to score thousands of free-text answers cheaply. That judge is itself a model, with documented failure modes. Zheng et al. (the MT-Bench paper) names four: position bias (favors whichever answer it saw first), verbosity bias (favors longer answers regardless of quality), self-enhancement bias (favors answers written in its own style), and limited reasoning ability (grades math and logic it cannot itself do).
LiveThe 80% bar - and what it really means4 min▶
The same paper found that GPT-4-class judges reach "over 80% agreement" with human raters. Sounds mediocre - until you learn the bar: 80% is also the human-human agreement rate. Two trained humans grading the same answers disagree about one time in five. So the best judges agree with humans about as often as humans agree with each other - and no judge can meaningfully beat that ceiling, because the ceiling is the noise in the task itself.
- A vendor claiming "95% judge accuracy" against human labels deserves a hard question, not applause - what got simplified until agreement looked that easy?
- A judge at 80% agreement with humans is a GOOD judge. A judge with no reported agreement number is an unmeasured one.
- The leader rule, verbatim, for your next review: never accept a judge score without its human-agreement number. A judge score is a model output, not a measurement - it becomes a measurement only after it is checked against humans.
Self-studyWhat the paper does about the biases2 min read▶
Zheng et al. do not stop at diagnosis - the paper's own mitigations are the ones to look for in any team's judging setup:
- Against position bias: swap the answer order and keep only the verdicts that stay consistent both ways - a judge that flips with the order gets no vote on that pair.
- Against verbosity bias: exactly what tonight's demo does - measure the judge against human labels, find the length pattern, and control for it.
The pattern behind both: the fix is procedural, not a better judge model. Ask a team "how do you control for the judge's biases?" and listen for a procedure, not a model name.
Bigger was less truthful. TruthfulQA - 817 adversarial questions across 38 categories, built to trigger learned falsehoods - made headlines at publication: the best model was truthful on just 58% of questions versus 94% for humans, and LARGER models were LESS truthful, because scale made them better imitators of popular human falsehoods. The inverse-scaling headline carries the leader lesson: capability is not honesty. A model that writes beautifully can still be confidently wrong, and no amount of "it's the newest model" settles a truthfulness question - only measurement does.
Catch the judge's bias, live ★ 24 min · watch + discuss
Below is an LLM-as-judge calibration check - 24 answers, each with a fixed human label, graded by a scripted judge that carries the verbosity bias documented in Zheng et al. You do not need to run anything technical: two buttons, one lesson. This is the exact exercise you should demand from any team proposing to grade quality with a judge.
Press Raw judge. Agreement with the 24 human labels: 58%. Look at the shape of the misses - 10 bad-but-long answers rated good. Every failure is the same failure: length is being mistaken for quality.
Say the diagnosis in Part 2's vocabulary: verbosity bias. Nobody tampered with this judge - it is doing what unmeasured judges do.
Press Length-controlled judge. Agreement jumps to 88%, with 1 residual miss - a genuine borderline call, the kind no grader escapes. Note that 88% is at the human-human bar: this judge is now as good as a judge gets.
The leader takeaway, out loud: you MEASURE a judge before trusting it. The 58%-to-88% move cost no new model and no budget - it cost one calibration check against human labels. Teams skip it because nobody asks. You now ask.
Close with the room's own dashboards: which of your current LLM quality numbers has been through this exercise? Any score that has not is a raw-judge score until proven otherwise.
Try it yourself - this week ◐ 20 min total
- Pull up one LLM quality report your organization already receives. For each row, write the plain-English question it answers - and mark any row where you cannot tell which question it is.
- Ask the owning team the hallucination question: measured against the world, or against the document? Note whether the answer comes quickly.
- Ask for the judge's human-agreement number on your main quality metric. If it does not exist, you have found this quarter's cheapest quality investment - a one-day calibration check.
- Optional depth: the practitioner track runs the faithfulness formula and the judge calibration hands-on in b6 and b7 - forward those to your technical leads.
Official sources covered
The leader track distills the canonical sources into decisions - definitions and formulas stay with the linked originals, and the practitioner track works them hands-on. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your RAG assistant states a fact that is TRUE in the real world but appears nowhere in the retrieved document. What is this?
Faithfulness measures support-by-the-document, not truth-about-the-world. An ungrounded answer that happens to be true today may be false next time - the system has stopped being constrained by its sources, which is exactly what faithfulness exists to catch.
2 · A GPT-4-class judge reaches "over 80% agreement" with human raters. What does that number mean?
Two trained humans disagree about one time in five on this kind of grading, so 80% IS the ceiling - the noise lives in the task, not the judge. A judge at the human-human bar is a good judge; a claimed 95%+ deserves scrutiny, not celebration.
3 · A team presents "quality: 4.3/5, scored by our LLM judge." What do you ask for before accepting it?
The demo's whole arc: the raw judge scored answers confidently at 58% agreement, and only measuring it against humans exposed the verbosity bias. A frontier judge or a second judge can share the same bias - only the human-agreement number tells you what the score is worth.