One call, four columns, a diagnosis
Individual metrics answer individual questions. A metric suite answers the real one: where is my RAG pipeline weakest? RAGAS lets you score retrieval and generation in a single run and lays the numbers side by side, so a low faithfulness points at the generator while a low context recall points at the retriever. Today we assemble a dataset over Recall, run the suite, and practice the reading skill that turns four numbers into a fix list.
The suite + a sample 9 min live
RAGAS runs on a dataset with a fixed shape. Each sample carries the question, the answer the system produced, the contexts it retrieved, and optionally a ground_truth reference. You pass that dataset and a list of metrics to evaluate(), and it returns a score per metric averaged over the samples - plus a per-row table you can mine for the worst answers.
LiveThe dataset shape3 min▶
Everything in RAGAS starts from the sample. Get the four fields right and the rest is a function call.
- user_input - the question asked.
- response - the answer the system generated.
- retrieved_contexts - the list of chunks the retriever returned for that question.
- reference - the ground-truth answer. Optional, but required by any metric that checks correctness or completeness.
Which metrics need that reference column is the thing to internalize now, because it decides whether a metric can run on live traffic or only on a golden set.
LiveThe four core metrics, together3 min▶
The workhorse RAGAS suite pairs two generation metrics with two retrieval metrics, so a single run covers both halves of the pipeline.
- Faithfulness (generation) - claims supported by context / total claims. Hallucination check.
- Response relevancy (generation) - mean cosine of N generated questions vs the input. On-topic check. Formerly Answer Relevancy.
- Context precision (retrieval) - ranking quality of the retrieved chunks: are the relevant ones near the top?
- Context recall (retrieval) - completeness: did retrieval bring back everything the reference answer needed? Needs the ground truth.
Self-studyBuild a dataset and run evaluate()4 min read▶
Here is the whole thing over three Recall answers: assemble the samples, wrap them in an EvaluationDataset, and call evaluate() with the four metrics. Note that context recall gets a reference and the two generation metrics do not need one.
to_pandas() tells you which questions failed and why. Sort by the weakest column and read the three worst rows - that is where every improvement in b7 and b8 starts.
The Nvidia dual-judge trio + reading results 8 min live
RAGAS also ships a trio of Nvidia metrics designed for stability: each uses two LLM judges on a small integer scale, which is more robust than a single judge on a fine-grained scale. Then we practice the real skill of this whole track - reading a results table as a diagnosis of where the pipeline breaks.
LiveThe Nvidia dual-judge trio3 min▶
These three metrics were designed for reliability: each uses two LLM judges and scores on a small integer scale, which is less noisy than asking one judge for a fine-grained number.
- Context Relevance - how relevant the retrieved context is to the query, scored {0, 1, 2} out of 2. A retrieval-quality signal.
- Response Groundedness - how well the response's claims are supported by the retrieved context, scored {0, 1, 2} out of 2. The trio's version of faithfulness.
- Answer Accuracy - how well the response matches the ground truth, scored {0, 2, 4} out of 4. This one needs a reference. It is the correctness signal we go deep on in b7.
LiveReading results as a diagnosis3 min▶
The payoff of running metrics together is component-wise diagnosis. Because the suite scores retrieval and generation separately, a low number points at a specific stage.
- Low faithfulness or groundedness means the answer says things the context did not support. The retriever did its job; the generator is the problem - fix the prompt or the model.
- Low context recall means the right information never made it into the context. The generator cannot ground on what it never received; the retriever is the problem - fix chunking, embeddings, or search.
- Low context precision means the relevant chunks are buried under noise - a ranking problem in the retriever.
- High faithfulness but low answer accuracy means the answer is grounded in a context that is itself wrong or outdated - a knowledge-base problem, not a model one.
The team that tuned the wrong half. A group (anonymized) saw a mediocre RAG scorecard and spent two weeks swapping generation models, with no improvement. When they finally read the columns, faithfulness was fine at 0.9 while context recall sat at 0.5 - the retriever simply was not finding the right chunks. Two days of chunking and embedding work moved the whole system more than two weeks on the generator. The results table had said so on day one.
Assemble a dataset and predict the lowest metric ★ 12 min · pen and paper, then code
Predicting a score before you run it is how you build intuition. Guess first, then let RAGAS check you.
Assemble three samples. Pick three Recall answers across corpora A, B, and C. For each, write out user_input, response, retrieved_contexts, and a reference ground truth. Keep them honest - include at least one answer you suspect is weak.
Read each sample like the judge will. For every sample ask: does the answer add anything the context did not say (faithfulness risk)? Did the retriever miss a chunk the reference needed (context recall risk)? Is the answer on topic (relevancy)?
Predict the lowest metric. Across the three samples, write down which of the four metrics you expect to score lowest, and one sentence of why. This is the prediction you are testing.
Reflect on the diagnosis. If your predicted-lowest metric is a generation metric, your fix list points at the prompt or model; if it is a retrieval metric, it points at chunking and search. In one line: which half of the pipeline would you invest in next, and does the metric agree?
Before session b7 ◐ 40 min total
- Build a RAGAS EvaluationDataset of at least five samples from your own system (or Recall) and run evaluate() with faithfulness, response relevancy, context precision, and context recall.
- Call
to_pandas(), sort by the weakest column, and write a one-line diagnosis for each of the three worst rows - generation problem or retriever problem? - Add the Nvidia Answer Accuracy metric to the run and note how its {0, 2, 4} scale behaves next to the 0-to-1 metrics.
- Read: the RAGAS metrics docs (core metrics + the Nvidia metrics page) and revisit b2 on testset generation as your source of ground truths.
Official sources covered
Taught from the RAGAS documentation. This page covers ~80% of its working content on running the suite - answer correctness gets its own deep dive in b7.
Three questions before you go 🎯 ◐ 90 seconds
1 · A RAGAS EvaluationDataset sample carries which fields?
Each sample pairs user_input, response, and retrieved_contexts, plus an optional reference. Metrics that check completeness or correctness - like context recall - need that reference column.
2 · Recall's scorecard shows faithfulness 0.9 but context recall 0.5. The most likely problem is...
High faithfulness means the generator grounds on the context it receives. Low context recall means the right information never reached it. That splits the diagnosis onto the retriever - fix chunking, embeddings, or search, not the prompt.
3 · The Nvidia trio (Context Relevance, Response Groundedness, Answer Accuracy) is designed to be more stable because...
The trio uses dual judges on small integer scales - Context Relevance and Response Groundedness on {0,1,2}/2, Answer Accuracy on {0,2,4}/4 - which is less noisy than a single judge assigning a fine-grained score.