Two metrics is not enough
Hit Rate says "did the right doc show up at all". MRR says "how high did the first right doc rank". Real questions need more: how much of the top k is actually relevant (precision), did we find all the relevant docs (recall), and how good is the whole ranking when relevance is graded rather than yes/no (NDCG). This session gives you the family, the formulas, and a rule for picking the one metric you will actually hold your retriever to.
Precision, recall, and where the metrics read from 9 min live
Every retrieval metric reads from the same object: a ranked list of results, each either relevant or not for this query. Once you can see that list, every metric is just a different question asked of it. Here is one ranked list, and the four metrics pointing at what they measure.
LiveThe formula table3 min▶
Four metrics, four formulas, four different failures they catch. Keep this table next to the ranked list above.
| Metric | Formula | What it catches |
|---|---|---|
| Hit Rate@k | fraction of queries with a relevant doc in top k | Did we find it at all, anywhere in top k |
| MRR | mean of 1/rank of the first relevant doc (0 if none) | How high the first right answer ranks |
| Precision@k | relevant in top k / k | Noise: how much of what we returned was junk |
| Recall@k | relevant in top k / total relevant | Gaps: how much relevant material we missed |
LiveContext precision vs context recall3 min▶
RAGAS frames these two as the retriever's twin responsibilities, and names them in a way worth borrowing.
- Context precision = ranking quality. It measures whether the relevant chunks are ranked high among what was retrieved. A retriever can return the right chunk buried at rank 8 and still have poor context precision - the good stuff has to be near the top.
- Context recall = completeness. It measures whether all the chunks needed to answer were retrieved. Crucially, context recall requires a reference (a ground-truth answer or the set of relevant chunks) to know what "all" means - you cannot compute recall without knowing the full set of relevant docs.
- They trade off. Widen k and recall rises while precision usually falls (more relevant found, but more junk too). The right balance depends on the product, which is Part 2's whole subject.
Read the scorecard as formulas ★ 8 min live
This is the required build-along. Below is Recall's golden set scored live - the same numbers as b1, but this time you connect each on-screen number to the formula behind it. Move k and watch Hit Rate and Precision@1 diverge: they answer different questions and respond to k differently.
Live★ Build-along: numbers to formulas6 min▶
For the current k, take one row of the table and compute the metric by hand from the formula table above, then confirm it matches the on-screen number. Then change k: watch Hit Rate climb (a wider net finds more) while Precision@1 stays fixed (position 1 does not care about k). That divergence is the whole point.
Tie it back to the formulas: Hit Rate@k is hits anywhere in top k / number of queries, so raising k can only hold or raise it. Precision@1 is relevant in top 1 / 1, so it is decided entirely by what sits at rank 1 - untouched by k. MRR reads only the first hit's rank, so it too ignores k past that first hit.
NDCG, and choosing the one metric to gate on 9 min live
Hit Rate, MRR, and Precision treat relevance as yes/no. But some results are more relevant than others, and their position matters. NDCG handles both: graded relevance, discounted by rank. It is the metric of choice when the whole ordering matters, not just the first hit.
Self-studyThe full suite from LlamaIndex3 min read▶
You do not implement these one by one in production. LlamaIndex's evaluator takes the metric names and reports the whole family over your golden set.
One call, the whole family. Now the question is not how to compute them - it is which one you actually hold the retriever to.
Self-studyMRR vs NDCG, and which metric for which product3 min read▶
The last thing that matters is fit: the right metric depends on what your product does with the results.
- MRR vs NDCG. MRR cares only about the rank of the first relevant result and treats relevance as yes/no. NDCG grades relevance and scores the whole ranking. Use MRR when one good hit is enough; use NDCG when the ordering of several results matters.
- One-answer bot → Precision@1 or MRR. If the product shows a single answer (a support bot, a "top result" assistant), you only care about position 1. Gate on Precision@1 or MRR and ignore how the tail is ordered.
- Research tool → Recall or NDCG. If the product surfaces many results for a human to scan (a search tool, a research assistant), completeness and whole-ranking quality matter. Gate on Recall (did we surface everything relevant) or NDCG (is the whole ordering good).
Choose Recall's gating metric ★ 10 min · the scorecard above
A retriever with five metrics and no chosen gate has no gate at all. Decide the one number Recall lives or dies by.
State the product. Write one line: what does Recall do with retrieved results? One grounded answer per query, or a list a human scans? Be honest about today, not the roadmap.
Map product to metric. One-answer → Precision@1 or MRR. Many-results → Recall@k or NDCG. Pick the single metric your product profile points at.
Read it off the scorecard. Find your chosen metric's current value in the live scorecard above. That number, at your chosen k, is the candidate gate.
Set and justify a threshold. Write the gate: "ship only if [metric] ≥ [value] at k=[k]" and one sentence on why that bar fits the product. That line is what b8's CI gate will enforce.
Before session b4 ◐ 45 min total
- Run LlamaIndex's RetrieverEvaluator with all five metric names over the golden set you grew in b2, and record the averaged numbers.
- By hand, compute DCG@3 and NDCG@3 for one query with graded relevance 3, 2, 1 at ranks 1, 2, 3 - confirm the perfect ordering gives NDCG = 1.0.
- Write your one-line gating rule for your own retriever, with the metric, the k, the threshold, and one sentence of product justification.
- Read: RAGAS context precision and context recall docs, LlamaIndex's retrieval evaluation guide, and the DCG/NDCG entry on Wikipedia.
Official sources covered
Taught from official docs. This page covers ~80% of their working content on retrieval metrics - the rest (custom metrics, hosted runners) lands in later sessions.
Three questions before you go 🎯 ◐ 90 seconds
1 · Precision@k and Recall@k differ because...
Precision divides by k and catches noise; recall divides by the total number of relevant docs and catches gaps - which is why recall needs a reference to know that denominator.
2 · NDCG differs from MRR mainly because...
MRR reads only the first relevant result's rank with binary relevance. NDCG grades relevance and discounts by rank across the whole list, then normalizes to 0-1 by the ideal ordering.
3 · For a support bot that shows the user one answer, the best gating metric is...
When the product shows a single answer, only position 1 matters. Precision@1 or MRR captures that; recall and wide-k metrics reward a tail the user never sees.