Two answers, both retrieved perfectly, one still wrong
A retriever can hand the model the exact right chunk and the model can still blow it: it invents a detail the chunk never mentioned, or it wanders off and answers a slightly different question. Retrieval metrics are blind to both - the chunks were right. Generation metrics grade the sentence the user actually reads. Today we add four of them to Recall, our RAG assistant over corpora A, B, and C, and we are careful about the one trap that catches everyone: these metrics measure faithfulness and focus, not truth.
Faithfulness + relevancy 9 min live
Two metrics, two different questions about one answer. Faithfulness asks: is every claim in the answer actually backed by the retrieved context? Response relevancy asks: does the answer stay on the topic the user asked about? Neither one asks whether the answer is true - that is a separate metric (b7). Getting this distinction into your bones is most of today.
LiveFaithfulness: claims supported divided by total claims3 min▶
RAGAS computes faithfulness in two LLM passes. First it breaks the answer into individual factual claims. Then, for each claim, it asks: can this be inferred from the retrieved context? The score is simply the fraction that can.
- The formula. Faithfulness = (claims supported by context) / (total claims). All three claims backed gives 1.0; the answer that adds one invented detail to two good ones scores 0.67.
- It is a hallucination detector. A low score means the model said something the context did not support - the textbook definition of a RAG hallucination. This is the single most useful generation metric for a support bot.
- No reference needed. Faithfulness compares the answer only to the context that was retrieved. You do not need a gold answer, which makes it cheap to run on live traffic.
- It is not accuracy. An answer can be perfectly faithful to a context that is itself wrong or outdated. Faithful means "grounded in what we gave it", not "correct".
LiveResponse relevancy: N questions back from the answer3 min▶
Response relevancy (renamed from Answer Relevancy in RAGAS v0.2+) measures whether the answer actually addresses what was asked. The clever trick: it works backwards. An LLM reads the answer and writes N questions (default 3) that this answer would be a good reply to. Each generated question is embedded and compared by cosine similarity to the real question. The score is the mean.
- The intuition. If the answer is on topic, the questions you can reconstruct from it look like the question that was asked. If the answer rambled, the reconstructed questions drift away and cosine similarity drops.
- It catches off-topic, incomplete, and evasive answers. A response that dodges the question, over-answers, or trails into a tangent scores low even when every sentence is faithful.
- It measures on-topic, NOT accuracy. A confidently wrong answer that stays squarely on the question can still score high on relevancy. Relevancy is about focus, not correctness.
- No reference needed either. Like faithfulness, it uses only the answer and the original question, so it runs on production traffic.
Self-studyScoring Recall's answers with RAGAS4 min read▶
In RAGAS you assemble a dataset of samples - each with the question, the model's answer, and the retrieved contexts - then hand a list of metrics to evaluate(). Faithfulness and ResponseRelevancy need no reference, so this runs on any sample of Recall's live answers.
The manager-approval sentence has no support in the context, so this sample's faithfulness lands below 1.0 while its relevancy stays high - the answer is on topic, it just added a detail. That signature, high relevancy plus dented faithfulness, is exactly a hallucination.
Noise sensitivity + semantic similarity 8 min live
Two more metrics for two more failure shapes. Noise sensitivity asks how easily irrelevant retrieved context knocks the model into a wrong claim. Semantic similarity asks how close the answer is in meaning to a known-good reference - forgiving different wording where an exact-match check would not.
LiveNoise sensitivity: how easily noise knocks it wrong3 min▶
Real retrieval hands the model a mix - some relevant chunks and some near-miss noise. Noise sensitivity measures how often that noise leaks into a wrong claim. It is the fraction of claims in the answer that are incorrect, given the ground truth.
- The formula. Noise sensitivity = (incorrect claims) / (total claims). Because it counts mistakes, lower is better - the opposite direction from faithfulness and relevancy. A robust model reads past the noise and scores near 0.
- It needs ground truth. To label a claim incorrect you need to know the right answer, so this is an offline metric you run against a golden set, not on raw live traffic.
- When it is useful. Reach for it when your retriever returns a lot of borderline chunks and you want to know whether the generator is disciplined enough to ignore them. It isolates a robustness problem that faithfulness alone can miss.
LiveSemantic similarity: meaning, not wording3 min▶
Sometimes you have a reference answer and want to know how close the model got - but exact string matching punishes every valid paraphrase. Semantic similarity embeds both the answer and the reference and reports the cosine similarity between the two vectors.
- The formula. Semantic similarity = cosine(embed(answer), embed(reference)). 1.0 means the same meaning; values near 0 mean unrelated. It is smooth and forgiving of rewording.
- It needs a reference. There must be a known-good answer to compare against, so it belongs to your offline golden-set evaluation.
- When it is useful. A quick, cheap proxy for "did the answer land near the target meaning" when you have references but do not want a full LLM-judge correctness call. It is a component of answer correctness in b7, not the whole thing.
- The caveat. High similarity is not the same as correct. Two answers can be semantically close yet disagree on the one fact that matters, and embeddings will not always separate them.
The paraphrase that failed a string check. A team (anonymized) graded a support assistant with exact-match against a reference answer and watched scores crater - not because the bot was wrong, but because it wrote "we will refund you within a week" where the reference said "refunds post in 5-7 business days". Swapping to semantic similarity recovered the signal: cosine 0.93, clearly the same meaning. Exact match had been measuring wording, not helpfulness.
Score a faithful vs an unfaithful answer by hand ★ 10 min · pen and paper
Do the decomposition yourself once and faithfulness stops being a black box. Take one Recall question and two candidate answers to it.
Pin the context. Write out the one or two retrieved chunks Recall was given for the question - say the refund policy chunk: "Refunds are issued to the original card within 5-7 business days."
Decompose answer A into claims. Answer A: "Cancel in Settings, then your refund posts to your original card in about a week." Break it into atomic claims: (1) cancel in Settings, (2) refund goes to original card, (3) takes about a week. Check each against the context. All three are supported. Faithfulness = 3/3 = 1.0.
Decompose answer B into claims. Answer B: "Cancel in Settings, refunds post in 5-7 days, and amounts over $500 need manager sign-off." Claims: (1) cancel in Settings - supported, (2) 5-7 days - supported, (3) over $500 needs sign-off - NOT in the context. Faithfulness = 2/3 = 0.67.
Reflect. Both answers are on topic, so response relevancy would score both high. Only faithfulness separates them - and the invented sign-off rule is exactly the kind of confident hallucination that gets a support bot in trouble. In one line: which metric would have caught it, and would it have caught it without a reference answer?
Before session b5 ◐ 40 min total
- Run RAGAS
Faithfulness()andResponseRelevancy()over five of your own system's answers (or five Recall samples). Find the answer with the biggest gap between the two scores and explain in one line what failure that gap describes. - Write one answer that is deliberately faithful but irrelevant (grounded in the context but not what was asked) and confirm relevancy drops while faithfulness stays high.
- Take a reference answer and reword it completely without changing its meaning; compute semantic similarity and note how far above an exact-match score it lands.
- Read: the RAGAS docs for faithfulness, response relevancy, noise sensitivity, and semantic similarity, and skim the TruLens groundedness leg of the RAG Triad.
Official sources covered
Taught from official docs. This page covers ~80% of their working content on generation metrics - the correctness metric that combines several of these lands in b7.
Three questions before you go 🎯 ◐ 90 seconds
1 · RAGAS faithfulness is computed as...
Faithfulness decomposes the answer into claims and scores the share that can be inferred from the context: supported / total. It is the hallucination detector, and it needs no reference.
2 · Response relevancy measures...
Relevancy has an LLM generate N questions from the answer and compares them by cosine to the original question. A confidently wrong but on-topic answer can still score high, so it measures focus, not truth.
3 · For noise sensitivity, a lower score is better because...
Noise sensitivity is (incorrect claims) / (total claims) under noisy context, and it needs ground truth to label claims. Because it counts errors, lower is better - the reverse of faithfulness and relevancy.