What this page covers, and what it deliberately does not
Metric depth lives in two sibling courses and this one does not re-teach it. Learn Evals goes properly into retrieval and generation metrics, RAGAS internals and tracing. Model Evaluation covers the classic and LLM-era metric canon side by side, including the Goodhart trap. Go there to learn the metrics. Stay here for the only question this course cares about: which of these numbers can you put on a launch gate, and what exactly does the number mean when you do?
One formula wearing three different names 7 min live
Faithfulness, groundedness and hallucination rate are usually the same measurement reported three ways. Once you see the shared pipeline, the interesting question stops being "which metric" and becomes "who decides what counts as a claim".
claims = extract_claims(answer) # one fact per claim
supported = [c for c in claims
if entailed(c, retrieved_context)] # NLI verdict
faithfulness = len(supported) / len(claims) # RAGAS, 0 to 1LiveStep 1 is the whole metric, and nobody audits it4 min▶
Every number in this family is a fraction, and a fraction is only as stable as its denominator. The denominator is "total claims" - which is decided by a claim extractor, usually another language model, using a prompt you did not write.
Take one Beacon answer: "Refunds are processed within 5 business days, and you can track them in the billing portal." A coarse splitter sees one claim. A fine splitter sees three: refunds are processed; the window is 5 business days; tracking happens in the billing portal. If the 5-day figure is the wrong one and everything else is right, the coarse splitter scores 0.00 and the fine splitter scores 0.67. Same answer, same context, same model, same error - a two-thirds swing in the metric from a prompt nobody reviewed.
The practical consequence for a launch gate: freeze the extractor and version it. Treat a change to the claim splitter exactly as you would treat a change to the model - it re-baselines every historical number you have, and a quarter-over-quarter comparison across a splitter change is meaningless.
Self-studyGroundedness on an ordinal scale, and why you cannot average it3 min read▶
Some platforms report the same idea as an ordinal judge score rather than a fraction. Azure AI Foundry's groundedness evaluator runs on a 1 to 5 scale, where 5 means the facts in the response are verifiable in the provided source. Azure's agent evaluators use the same 1-5 shape with a default pass threshold of 3.
An ordinal scale is not a rate, and this trips up gates constantly. The distance between a 1 and a 2 is not the distance between a 4 and a 5, so an average of 4.1 tells you very little: it is equally consistent with "everything scored 4" and with "most things scored 5 and eight answers were fabrications scored 1". Gate on the distribution, not the mean - for example, "no case below 4 on the policy slice" rather than "mean above 4". Session b4 builds the rubric discipline that makes ordinal judge scores trustworthy in the first place.
The definition moves the number roughly tenfold 6 min live
This is the single most useful fact on the page, and the one that changes how you read every vendor claim afterwards. Two published scoring definitions were run over the same leaderboard. Not different models, not different prompts - the same outputs, scored twice.
What this does to a procurement conversation. Two vendors present. The first reports a 2% hallucination rate, the second reports 9%. The instinct in the room is that the first bot is four times better. The honest reading is that you have no idea, because you have been handed two numbers produced by two scorers with different standards for what counts as unsupported. The only comparison worth making is on your data with one scorer, and the only question worth asking in the meeting is which definition produced the figure on the slide. Session a3 turns this into a list of grilling questions for leaders; here it becomes a line in your eval config.
Self-studyWhy stricter definitions always look worse, and why that is fine2 min read▶
A strict scorer marks a claim unsupported unless the context entails it outright. A lenient scorer accepts plausible elaboration, common knowledge, and paraphrase that goes slightly beyond the source. Real answers are full of small elaborations, so the strict scorer finds far more of them. Neither is wrong; they are answering different questions.
The decision you actually have to make is which question matches your blast radius. Beacon's support bot states refund policy to paying customers, so "plausible elaboration" is exactly the failure mode that costs money - it should be scored strictly. An internal Insight Bot summarising a dashboard for a weekly review can tolerate a looser standard, because a human reads every line before it travels. Pick the strictness from the tier, then never move it because the number came back ugly.
Relevancy, citations, verbosity - and their wrinkles 6 min live
Faithfulness only tells you the answer is supported. It says nothing about whether the answer addresses the question, whether the citations are usable, or whether the reply was four paragraphs where one would do. Four more metrics, each with a wrinkle you have to gate around.
qs = generate_questions(answer, n=3) # RAGAS default N = 3
sims = [cosine(embed(q), embed(user_input)) for q in qs]
relevancy = mean(sims) # cosine is -1..1, so this can exceed [0,1]Read what that is actually doing. RAGAS does not compare the answer to the question directly - it asks a model to reverse-engineer the questions the answer would be a good reply to, embeds them, and measures how close they land to the question the user really asked. Three questions by default. It is a genuinely clever construction, and it measures whether the answer is on topic, not whether it is correct. A confidently wrong answer to exactly the right question scores beautifully here, which is precisely why it is never a gate on its own.
| Metric | Catches | Completely misses | Gate it as |
|---|---|---|---|
| Faithfulness | Facts not present in the retrieved context | Answers that are supported but irrelevant, or that omit the key fact | Mean plus a per-case floor |
| Response relevancy | Off-topic, evasive and padded answers | Confident wrongness on exactly the right topic | Never alone. Pair with faithfulness |
| Groundedness 1-5 | Fabrication, at a resolution a human can review | Anything below the threshold that the mean hides | Distribution, not average |
| Citation pair | Citing nothing, and citing everything | Correct citations attached to a wrong conclusion | Both scores, both above the bar |
| Hallucination rate | Drift in fabrication over live traffic | The cases nobody sampled that week | A trend with a rollback trigger |
| Verbosity 0-5 | Four paragraphs where one would do | Everything about correctness | A UX gate and a cost gate at once |
Score one answer by hand, twice ★ 10 min · everyone builds
You cannot gate on a metric you have never computed manually. Take one real answer from your own bot - or one from Beacon's support bot - and run the pipeline with a pen. It takes eight minutes and it permanently changes how you read a faithfulness score.
Pick one answer where the bot got something subtly wrong, and paste it next to the retrieved context that was actually in the prompt. Not the knowledge base. The passages the retriever returned on that specific call. If you cannot recover them, that is your first finding of the day and b10 is about fixing it.
Split the answer into atomic claims, coarsely - one claim per sentence. Mark each supported or unsupported against the context only. Compute supported / total. Write the number down.
Now split it again, finely - one claim per assertable fact, so a sentence with a number, a timeframe and a location becomes three claims. Score it again. Write the second number down next to the first.
Compare. On most real answers the two numbers differ by 15 to 40 points, and the bot did not change between them. That gap is definition sensitivity at the scale of a single answer - the same thing the HHEM-versus-FaithJudge gap shows at the scale of a leaderboard.
Last, score the same answer for relevancy the RAGAS way, by hand: write down the three questions this answer would be a perfect reply to. If none of them is the question the user asked, you have found an evasive answer that a faithfulness gate would have passed happily.
Freeze Beacon's metric contract ★ 8 min · everyone writes
A metric contract is one small file that says, for each gate, which definition produced the number and which knobs are frozen. It exists so that six months from now nobody can improve a score by changing a prompt in the grader. Write yours in this session; it is shorter than you expect.
support_bot: # tier T3, can issue refund credits
faithfulness:
source: RAGAS - supported claims / total claims
splitter: claim_splitter_v3 # FROZEN. changing it rebaselines
strict: true # no credit for elaboration
gate: mean >= 0.90 and no single case below 0.60
groundedness:
source: Azure 1-5 ordinal, 5 = verifiable in the source
gate: no case below 4 on the policy slice
citations:
source: Bedrock CitationPrecision + CitationCoverage
gate: both >= 0.85 # never one without the other
relevancy:
source: RAGAS, N = 3 reverse-engineered questions
clamp: [0, 1] # cosine is -1..1
gate: reported, not blocking # never a gate on its ownWrite one block per metric you intend to gate on. If a metric is not on a gate, it does not belong in this file - it belongs on a dashboard. Contracts get long and stop being read.
For every metric, record the source definition by name and the version of anything that produces the number: the claim splitter, the judge model, the judge prompt, the embedding model. These are the four knobs that silently move scores.
Mark each gate as blocking or reported. Relevancy is reported, because it passes confident wrongness. Faithfulness is blocking. Being explicit here is what stops a green dashboard from being mistaken for a green gate.
Add the per-case floor, not just the mean. "Mean at or above 0.90" is satisfied by ninety perfect answers and ten fabrications. A floor closes that door, and on a T3 bot the floor matters more than the mean.
Commit the file next to your eval code and put it under review. A pull request that changes a threshold or a splitter version should be as visible as one that changes the model - because it moves the number by just as much.
Try it yourself - this week ◐ 45-60 min total
- Score ten real answers by hand at both splitting granularities and record the two numbers side by side. The spread across those ten pairs is your own definition-sensitivity figure, and it is far more persuasive in an internal argument than any published leaderboard.
- Find the most recent quality number your team reported to anyone outside the team, and write one sentence naming the exact definition that produced it. If you cannot write that sentence in under a minute, the number was not defensible when you reported it.
- Check whether your relevancy scores are clamped. Compute the minimum and maximum across your last eval run - a value below 0 is not a bug in your bot, it is cosine similarity doing what cosine similarity does, and a gate written as "relevancy above 0.7" will behave strangely near the edges.
- Write the metric contract for one bot you own. Version the claim splitter, the judge model and the embedding model explicitly, even if the versions are "whatever the library defaulted to in August" - naming the gap is the first step to closing it.
- Pick your strictness setting from the risk tier before you look at any scores, and write down the reasoning. In b9 you will find out whether you would have quietly loosened it once the result came in.
Official sources covered
Every formula and figure on this page traces to a primary source - vendor evaluation documentation, published leaderboards and papers catalogued in this course's source map. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A vendor reports 2% hallucination and a competitor reports 10% on the same benchmark family. What is the most likely explanation?
Definition sensitivity is the largest single source of variation in published hallucination numbers, and it is larger than most model differences. Until you know which scorer produced a figure, comparing two figures across vendors tells you about the scorers, not the models. The only fix is to run one definition over your own data, which is exactly what the metric contract in Demo 2 exists to pin down.
2 · Your CI gate is written as "RAGAS response relevancy at or above 0.7" and it occasionally sees a value of -0.12. Is this a bug?
RAGAS reverse-engineers N questions from the answer, embeds them, and averages their cosine similarity to the original input. Cosine similarity is defined on -1 to 1, so the mean of several of them is too. A wildly off-topic answer can genuinely produce a negative score. It is not a defect - it is the definition - but a gate that assumes a 0-to-1 range will misbehave, so clamp explicitly and say so in the contract.
3 · Why does AWS recommend using CitationPrecision and CitationCoverage together rather than picking one?
A bot that attaches one very safe citation and says little else scores near-perfect precision. A bot that staples every retrieved passage to every sentence scores near-perfect coverage. Both are useless to a customer trying to check a policy claim, and both pass a single-metric gate. The pair closes the loophole in each direction, which is why it is a pair and not an average - averaging would let a strong score on one side hide a failure on the other, exactly like the conjunctive gates in b9.