Where tonight fits
b1-b5 evaluated a model that outputs a number. Lumen's new assistant outputs SENTENCES - answers stitched from retrieved context - and yet the evaluation instincts transfer almost perfectly: metrics are still opinions about which mistakes matter, and the precision/recall pair from b1 comes back reborn at sentence level. Division of labor with the sibling course, said honestly: learn-evals owns the hands-on harness (golden sets, RAGAS pipelines, tracing, CI); this course owns the metrics - definitions, use cases, and the misconceptions that get systems shipped wrongly. b7 next session asks the uncomfortable follow-up: who grades the grader?
Faithfulness, done properly 8 min live
The headline metric of RAG evaluation asks one question: does the answer stick to the evidence it was given? RAGAS defines it exactly: break the response into claims, check each claim against the retrieved context, and score supported claims / total claims, on a 0-1 scale. The worked example in the RAGAS docs scores 0.5 - one claim right, one date wrong. A variant, FaithfulnesswithHHEM, swaps the LLM checker for Vectara's trained hallucination detector as the grader.
LiveSame idea, different ruler: Foundry's groundedness4 min▶
Microsoft Foundry (Azure AI) measures the same concept with a different instrument: Groundedness is judged on a 1-5 Likert scale by an LLM judge, with a pass threshold of 3. And their framing hands you a gift - it maps straight back to b1:
- Groundedness = the PRECISION aspect - nothing in the answer is fabricated beyond the context.
- Response Completeness = the RECALL aspect - nothing critical in the context was left out of the answer.
The b1 pair, reborn at sentence level. And notice what the RAGAS-vs-Foundry difference teaches: same concept, one scores a 0-1 claim ratio, the other a 1-5 judged scale with a pass line. Metric design is a CHOICE - when two dashboards disagree on "groundedness," check the ruler before blaming the model.
LiveCRITICAL: faithful is not correct4 min▶
Faithfulness is measured against the CONTEXT - not against reality. An answer that perfectly restates a stale or wrong retrieved document scores a flawless 1.0 while being flatly false. RAGAS keeps a separate metric for truth-vs-reality: Factual Correctness, scored against a ground-truth reference - a different metric, deliberately.
Grounded in the wrong context = confidently wrong with a perfect score. A support bot answered pricing questions from a deprecated price sheet the retriever kept surfacing. Faithfulness: 1.0, every claim supported. Customers were quoted prices two revisions old for a month. No generation metric could catch it - the failure lived in retrieval, and only a retrieval metric (or a human reading answers against reality) would see it. Hold that thought for Part 2.
Self-studyThe faithfulness pass in pseudocode2 min read▶
Strip away the library and the whole metric is a dozen lines - which is exactly why you should know it cold:
# what every faithfulness grader automates
claims = decompose(answer) # split into atomic, checkable claims
supported = 0
for claim in claims:
if is_supported(claim, context): # judged per claim vs the chunks ONLY
supported += 1
faithfulness = supported / len(claims) # tonight: 3 / 4 = 0.75Both function calls hide a judgment: decompose decides what counts as one claim, and is_supported decides what counts as evidence. In RAGAS both are performed by an LLM (or by Vectara's HHEM detector in the variant) - which is why b7 exists: the grader is itself a model, with a model's failure modes.
The rest of the family, and the two hallucinations 7 min live
Faithfulness never works alone. A RAG system has two halves - retrieval and generation - and the rest of the family exists to tell you WHICH half broke, plus one guard against the oldest failure: a beautiful answer to the wrong question.
LiveRelevancy + the retrieval metrics4 min▶
- Response Relevancy (RAGAS renamed it from Answer Relevancy) - is the answer on-topic for the question? On-topic is NOT accuracy: an answer can be perfectly relevant and factually wrong, or faithful and beside the point. Separate dials.
- Context Precision - of the chunks retrieved, how many were actually useful? Low = the retriever floods generation with noise.
- Context Recall - of the evidence needed, how much did retrieval actually fetch? Requires a reference answer to score. Low = the evidence never arrived, and no prompt engineering can fix that.
- Noise Sensitivity - how much does irrelevant retrieved material degrade the answer?
The diagnostic power is in the COMBINATION: low faithfulness with high context recall means generation is inventing; high faithfulness with low context recall means retrieval starved it (the stale price sheet). Localize first, then fix the right half.
LiveThe two hallucinations - world vs document3 min▶
"Hallucination rate" gets quoted as if it were one number. It is two different failures, measured by two different benchmarks:
| Benchmark | What it measures | Headline numbers | The fix it points to |
|---|---|---|---|
| TruthfulQA | Lying about the WORLD - parametric knowledge parroting popular falsehoods (817 adversarial questions) | At publication: best model 58% truthful vs 94% for humans - and larger models were LESS truthful (inverse scaling) | Better model, better prompting - retrieval will not save a model that believes the myth |
| HHEM leaderboard (Vectara) | Lying about the DOCUMENT - faithfulness in summarization, measured at temperature 0 | Leaders sit around ~2-3% hallucination rates | Better retrieval and grounding - this is the RAG-relevant number |
Different failure, different fix. A vendor quoting "under 3% hallucinations" is citing the document kind; your users asking general questions will meet the world kind. Ask which one the number means before it goes on a slide.
Score one Lumen answer by hand ★ 20 min · everyone scores
An analyst asks Lumen's assistant: "why did March revenue dip?" (the same March the statistics and experimentation courses dissected). The assistant retrieves two context chunks - the weekly revenue report and the campaign calendar - and produces a four-claim answer. You are the grader. Pen, paper, no model.
Decompose the answer into claims. Four atomic statements: revenue fell 12% vs February; paid traffic dropped after the sale ended; a March 8 site outage caused the dip; recovery began in the final week. One claim per checkable fact - if a sentence bundles two facts, split it.
Check each claim against the two chunks - only the chunks. Claims 1 and 4 are supported by the revenue report; claim 2 by the campaign calendar. Claim 3 appears in NEITHER chunk. Whether an outage actually happened is not the question - the context is the only admissible evidence.
Compute faithfulness. Supported / total = 3 / 4 = 0.75. That is the entire RAGAS formula - you just ran it.
Now score the OTHER dials. Response relevancy: high - every claim addresses the question asked. And here is the twist: suppose the March 8 outage really happened (the site log confirms it, the retriever just never fetched that log). Claim 3 is UNGROUNDED but TRUE - world knowledge leaking in.
Say what each family member would report. Faithfulness: 0.75, flags claim 3. Factual Correctness (vs ground truth): would PASS claim 3. Context Recall: low - the evidence needed was never retrieved. Verdict: this is a retrieval failure wearing a generation costume. Fix the retriever, not the prompt.
Try it yourself - this week ◐ 20-30 min total
- Take one real answer from any assistant you use, decompose it into claims, and score faithfulness by hand against whatever source it cites. Note every judgment call you had to make - that list is b7's syllabus.
- Explain faithful-vs-correct to a colleague in two sentences, using the stale price sheet story. If they can repeat it back, you own the distinction.
- Find one "hallucination rate" claim in a vendor deck or blog post and classify it: world (TruthfulQA-style) or document (HHEM-style)? Was the distinction stated?
- Optional bridge: skim the RAGAS metrics docs and match each metric to the family map from Part 2 - then let learn-evals run the pipeline for real.
Official sources covered
Tonight covers the canonical RAG quality metrics from RAGAS and Microsoft Foundry, plus the two hallucination benchmarks everyone quotes and few distinguish. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A Lumen answer scores faithfulness 1.0, yet the customer was told a price two revisions old. What happened?
Faithful means "sticks to the evidence given," not "true." Grounded-in-wrong-context is confidently wrong with a perfect score. RAGAS keeps Factual Correctness (vs ground truth) as a separate metric precisely because these come apart.
2 · A model repeats a popular myth in a closed-book chat, and a different model contradicts the report it was asked to summarize. Which benchmark measures which?
Two different failures share one word. TruthfulQA: 817 adversarial questions about world knowledge, where larger models were LESS truthful. HHEM: faithfulness to a given document in summarization, temp 0, leaders ~2-3%. Different failure, different fix - better model vs better retrieval.
3 · An answer scores HIGH faithfulness but Context Recall is LOW, and users say the answers keep missing the real cause. Which half of the RAG system broke?
High faithfulness says generation stuck to what it was given; low context recall says what it was given was not enough. That combination localizes the failure to retrieval - fix the retriever, and no amount of prompt tuning will substitute. Localize first, then fix the right half.