Why "it feels right" is a trap
A RAG system has a lot of moving parts, and each one can fail silently. Bad chunks, a retriever that misses, a reranker that hurts, a model that hallucinates confidently over perfect context. When the answer looks wrong, which part do you fix? Without measurement you are guessing, and guessing at a five-stage pipeline is how a week disappears. Evaluation turns "something is off" into "context recall dropped to 0.4 - the retriever is missing chunks". That is the whole game today.
Retrieval metrics vs generation metrics 10 min live
Every RAG metric answers one of two questions: did we fetch the right context? (retrieval) or did we write a good answer from it? (generation). Keeping them apart is the single most useful habit in RAG evaluation, because it tells you which half of the pipeline to fix. A low retrieval score points at chunking and search; a low generation score points at your prompt and model.
LiveThe retrieval side: can it find the right context?3 min▶
Retrieval metrics judge the chunks you fetched, before the model writes a word. If these are bad, nothing downstream can save you - a perfect model over the wrong context still gives a wrong answer.
- Hit rate. The blunt one: was a relevant chunk anywhere in the top-k? It ignores position. Great for a first pulse on the retriever.
- Context precision. Ranking quality - are the most relevant chunks ranked highest? A retriever that buries the right chunk at position 8 scores poorly here even if hit rate says "found it".
- Context recall. Completeness - of everything needed to answer, how much did we actually retrieve? This one needs a reference answer to know what "everything" was.
RetrieverEvaluator reports Hit Rate and MRR (mean reciprocal rank). MRR rewards ranking the relevant doc high: if the right chunk is always at position 1, MRR = 1.0; at position 2, each query contributes 1/2; and so on. Hit rate says "did we find it", MRR says "how near the top". Use both - they answer different questions about the same retriever.
LiveThe generation side: is the answer good and honest?3 min▶
Generation metrics judge the answer the model wrote, given the context it was handed. Two of them (faithfulness, relevancy) need no reference answer at all - a huge practical win.
- Faithfulness. The anti-hallucination metric. RAGAS computes it as supported claims / total claims: it breaks the answer into claims and checks how many are backed by the retrieved context. 1.0 means every statement is grounded.
- Answer relevancy (RAGAS now calls it Response Relevancy). It generates questions from the answer and measures their mean cosine similarity to the real question. High = the answer stays on-topic. Note the trap: it measures on-topic-ness, not factual accuracy - a confident wrong answer can still be "relevant".
- Answer correctness. The one that needs a ground-truth answer: a blend of factual F1 (claims that match) and semantic similarity to the reference.
The relevancy false comfort. A team (anonymized) shipped a bot scoring 0.9 answer relevancy and celebrated - until users complained. The answers were beautifully on-topic and factually wrong. Relevancy was high because it only checks "is this about the question". Faithfulness was the metric that would have caught it, and it was sitting at 0.5. Never read relevancy as accuracy.
Here is the whole family on one card - what each measures and, crucially, the failure it catches:
| Metric | What it measures | Failure it catches |
|---|---|---|
| Hit rate | A relevant chunk is somewhere in top-k | Retriever misses the answer entirely |
| MRR | How high the relevant chunk is ranked | Right chunk found but buried too low |
| Context precision | Ranking quality of retrieved chunks | Noise ranked above the good chunk |
| Context recall | Completeness vs the reference answer | Part of the answer was never retrieved |
| Faithfulness | Supported claims / total claims | Hallucination - claims not in the context |
| Answer relevancy | Answer stays on-topic for the question | Rambling, evasive, or off-topic answers |
| Answer correctness | Factual F1 + similarity to ground truth | Answer that is on-topic but simply wrong |
Building a golden set and running an eval 9 min live
A metric is only as good as the questions you run it on. A golden set (reference set) is a small, curated list of questions - each with a reference answer and, ideally, the reference contexts that should have been retrieved. You do not need thousands. Twenty to fifty questions that cover your real usage, including the ones that failed, will move you faster than any benchmark. The flow is always the same: build the set, run the metrics, read the scorecard.
Self-studyA RAGAS-style eval, end to end4 min read▶
RAGAS wraps all of these metrics behind one evaluate() call. You assemble a small dataset - question, the answer Recall produced, the contexts it retrieved, and the ground-truth answer - then pass the metrics you care about. Faithfulness and answer relevancy work without ground truth; context recall and correctness use it.
Read that last line the way a mechanic reads a dashboard. Faithfulness and relevancy are healthy - the generator is doing its job. Context recall at 0.42 is the alarm: the retriever is leaving out chunks the answers needed. You now know exactly where to spend the next hour, and you never had to guess.
LiveWhich metrics need a reference, which are self-judging3 min▶
This distinction decides how much labelling work you sign up for, so hold it clearly:
- Need a reference answer / ground truth: context recall, answer correctness, and reference-based context precision. These compare against something you wrote by hand - more work to build, but they measure completeness and correctness that self-judging metrics cannot.
- LLM-as-judge only (no reference): faithfulness and answer relevancy. A judge model extracts and verifies claims, or generates questions from the answer, and scores from that alone. Cheap to run on every query - even in production.
gpt-4o-mini or a Claude model) does the reading. For faithfulness it extracts the answer's claims and checks each against the context. For relevancy it generates plausible questions from the answer and compares them to the real one. The judge is a model grading a model. It is not perfect, but it is consistent and fast, which is what makes eval-on-every-change affordable.
Turn your failures into a golden set ★ 12 min
Back in b1 you saved three queries where retrieval returned the wrong chunk, and in b4 you collected more failures while tuning the retriever. Those failures are the most valuable eval questions you own - they are exactly where Recall is weak. Turn them into a golden set and score them.
Gather your failures. Pull the three failing queries from b1 and any from b4. For each, write the question, the correct reference answer (by hand), and the chunk(s) that should have been retrieved.
Run Recall on them. For each question, capture what Recall actually generated and the contexts it actually retrieved. This is your answer and contexts columns.
Score it. Run the RAGAS-style eval with faithfulness, answer relevancy, context precision, and context recall. Do the end-to-end vibe check first, then read the numbers.
Diagnose. In one line per failure: is this a retriever problem (low recall/precision) or a generator problem (low faithfulness)? Now you know what to fix - and you have proof, not a hunch.
Before session b10 ◐ 45 min total
- Grow your golden set to at least 15 questions that cover Recall's real usage, not just the failures. Include a few you are confident it answers well - a good eval set has passes too.
- Add LlamaIndex's
RetrieverEvaluatorto your run and record Hit Rate and MRR for the retriever alone. Compare MRR before and after the reranker you added in b5. - Deliberately break something - drop your top-k to 1, or disable the reranker - re-run the eval, and watch which metric moves. Confirm the metric points at the part you broke.
- Read: the RAGAS metrics docs (faithfulness, response relevancy, context precision/recall) and LlamaIndex's retrieval evaluation guide. Skim OpenAI's "Evaluate RAG" cookbook for the eval-loop framing.
Official sources covered
Taught from the RAGAS and LlamaIndex evaluation docs plus OpenAI's cookbook. This page covers ~80% of their working content on RAG metrics - the rest (hosted dashboards, paid keys) stays with the source.
Three questions before you go 🎯 ◐ 90 seconds
1 · Your RAG answer is wrong. Faithfulness is high (0.95) but context recall is low (0.4). What is the most likely problem?
High faithfulness means the answer is faithful to the context it got; low context recall means the retriever didn't fetch everything needed. That points at the retriever, not the generator.
2 · In RAGAS, faithfulness is computed as...
RAGAS faithfulness breaks the answer into claims and measures how many are supported by the retrieved context - it is the anti-hallucination metric.
3 · Which metric does NOT require a reference answer / ground truth?
Answer relevancy is self-judging - it generates questions from the answer and compares to the real one, no reference needed. Context recall and answer correctness both require ground truth.