learn-ai-evals-with-phoebe / Builder session 1 of 10
Learn Evals with Phoebe · Builder track · Session 1 of 10

Your first metric: a scorecard that moves

Evaluation stops being abstract the moment you watch a number change. This session builds the smallest real eval: a golden set of questions with known right answers, run against Recall's retriever, scored with Hit Rate and MRR. You will do it live in the browser first - drag k and watch the score move and the misses appear - then see the same thing in Python. By the end you can measure a retriever instead of trusting it.

🟢 Builder track Practitioners · some Python Live scorecard included Start here
0-3 · Welcome 3-18 · Concepts 18-40 · Build-along 40-45 · Q&A
Part 0

The smallest eval that is still real

Every evaluation, however fancy, is the same three moves: take questions whose right answer you already know (a golden set), run the system, and compare. The rest of the track adds better questions, better metrics, and automation - but this loop is the whole idea. We build it today over Recall, the RAG assistant from the RAG course, and we make the score move in your browser before a line of Python.

Live - presented in session Self-study - full depth after class ★ Try it now Sources covered at the end
★ What you build today A working intuition for a golden set, the two most common retrieval metrics (Hit Rate@k and MRR), hands-on time with a live scorecard over Recall, and the Python to compute the same numbers - the first real measurement of an AI system you will make.
Part 1 · covers the eval loop + the golden set

Golden set, run, compare 7 min live

A golden set is a list of questions paired with the answer (or the document) that should come back. You run the system on each question and check whether it got the right one. Count the hits and you have a metric. That is evaluation, stripped to its bones.

Golden set question + expected answer System under test Recall retrieves Compare got == expected? Score count the hits Every eval in this course is this loop. The metrics just change how "compare" turns into a number.
🔍 Click to zoom - the eval loop: golden set, run, compare, score
LiveWhat makes a set "golden"3 min▶

Golden just means "we agreed on the right answer in advance". For a retriever, the right answer is usually "which document should come back for this question". Our golden set has twelve questions across Recall's three corpora, each tagged with the one chunk that should rank first.

  • Known answers are the whole point. Without a right answer to compare against, you are back to vibes. The golden set is where human judgment gets captured once, so the machine can check against it forever.
  • Small but representative beats big but skewed. Twelve honest, varied questions teach you more than a thousand near-duplicates. Session b2 is entirely about building these well.
  • It doubles as a regression net. Once the set exists, every future change gets re-scored against it automatically. Today's twelve questions are the seed of the CI gate in b8.
LiveHit Rate and MRR, without the jargon3 min▶

Two metrics answer the two questions you actually have about a retriever: did the right document come back at all, and how high did it rank?

MRR by hand: three questions, ranks 1, 2, and 3 Q1 - rank 1 1.00 Q2 - rank 2 0.50 Q3 - rank 3 0.33 MRR (average) 0.61 (1 + 0.5 + 0.33) / 3 = 0.61 - only the first correct hit counts, not later ones.
🔍 Click to zoom - MRR rewards a right answer at rank 1 far more than at rank 3
  • Hit Rate@k = the fraction of questions where the right document showed up in the top k results. "@3" means "somewhere in the top 3". It answers did we find it.
  • MRR (Mean Reciprocal Rank) = the average of 1 divided by the rank of the right document. Right answer at position 1 scores 1.0; at position 2, 0.5; at position 3, 0.33. It answers how high did it rank - and only the first correct hit counts.
  • Rank over threshold. Early on, trust the ranking (and these rank-based metrics) more than any absolute similarity score, which drifts by model.
The MRR formula, once MRR = average of (1 / rank of the first right answer) across all questions, counting 0 when the right answer never appears. Example: three questions with the right answer at ranks 1, 2, and 3 gives (1 + 0.5 + 0.33) / 3 ≈ 0.61. High MRR means the right thing is not just present but near the top.
Part 2 · covers the live scorecard

Watch the score move 8 min live

Numbers on a slide are forgettable; a number you can move is not. Below is a real evaluation running in your browser - twelve golden questions against Recall's retriever. Change k and watch Hit Rate climb, MRR hold steady (it does not care about k past the first hit), and the misses turn to hits.

Live★ Build-along: your first scorecard6 min▶

Click k=1, then k=3, then k=5. Watch Hit Rate rise as the net widens, watch which questions flip from miss to hit, and notice Precision@1 (right answer ranked first) stay put - because widening k cannot change what is already at position 1.

Honesty note - what this scorecard really does Retrieval here uses the same simplified lexical embedder as the RAG course playground, so it runs with zero network. But the evaluation math - Hit Rate@k, MRR, Precision@1 over a golden set - is exactly what you compute in production. You are watching the real metric mechanic with a toy retriever.

Read the table underneath the bars: each row is a golden question, its expected chunk, what actually ranked first, the rank the right answer landed at, and whether that counts as a hit at the current k. That table is a bug report and a to-do list in one - every miss is a retrieval improvement waiting to happen.

Self-studyThe same numbers in Python4 min read▶

Outside the browser the loop is identical: for each golden question, retrieve, find the rank of the expected id, and roll up. Here it is from scratch, then the one-liner the ecosystem gives you.

Python · Hit Rate + MRR from scratchgolden = [ {"q": "cancel and get money back", "expected": "C-refund"}, {"q": "I forgot my password", "expected": "C-reset"}, # ... your full golden set ] def evaluate(retriever, golden, k=3): hits, rr = 0, 0.0 for item in golden: ranked = retriever(item["q"]) # returns chunk ids, best first if item["expected"] in ranked[:k]: hits += 1 if item["expected"] in ranked: rank = ranked.index(item["expected"]) + 1 # 1-based rr += 1 / rank n = len(golden) return {"hit_rate@%d" % k: hits / n, "mrr": rr / n} print(evaluate(recall_retriever, golden, k=3))

And with LlamaIndex, the same metrics come from a built-in evaluator - you supply the golden set and it reports hit rate and MRR (plus precision, recall, NDCG) for you.

Python · LlamaIndex RetrieverEvaluatorfrom llama_index.core.evaluation import RetrieverEvaluator evaluator = RetrieverEvaluator.from_metric_names( ["hit_rate", "mrr"], retriever=retriever ) result = evaluator.evaluate(query="cancel and get money back", expected_ids=["C-refund"]) print(result) # hit_rate + mrr for this query; use evaluate_dataset for the whole set
Why rank-based metrics first Hit Rate and MRR need only a list of ids and a known-right id - no LLM, no reference text, cheap and deterministic. They are the fastest signal that a retrieval change helped or hurt, which is why every regression suite (b8) starts here before adding the pricier generation metrics.
Build-along · take it further

Read the scorecard like a to-do list ★ 10 min · the scorecard above

A scorecard is only useful if it changes what you do next. Mine the one above for actions.

Find the misses. At k=1, note every question whose right answer was not ranked first. Write down what actually came back instead.

Diagnose one. Pick a miss and read the expected chunk. Why did the retriever prefer something else - shared words with a wrong chunk, a vague question, a genuinely ambiguous case? This instinct is the whole of retrieval tuning.

Watch MRR vs Hit Rate. Note a question that is a "hit" at k=5 but ranked 5th. Hit Rate calls it a win; MRR barely rewards it. Which metric would you trust for a system that only shows users the top result?

Same hit, two different verdicts: rank 5 at k=5 HIT RATE@5 SAYS: WIN Right answer ranked 5th, counts as a full hit MRR SAYS: BARELY Contributes only 1/5 = 0.20 Same signal as almost missing it For a UI that shows one answer, trust MRR - Hit Rate cannot tell rank 1 from rank 5.
🔍 Click to zoom - Hit Rate cannot tell rank 1 from rank 5; MRR can

Reflect. In one line: is this retriever good enough to ship for a support bot that shows one answer? Your k=1 Precision is the number that decides.

★ Recall's scorecard after b1 Recall now has a number, not a vibe. You can measure its retriever and spot every miss. In b2 we make the golden set itself trustworthy - because a lazy golden set produces confident, meaningless scores.
Homework

Before session b2 ◐ 40 min total

Source material

Official sources covered

Taught from official docs. This page covers ~80% of their working content on first-metric retrieval evaluation - the rest (hosted runners, full metric suites) lands in later sessions.

✓LlamaIndex · retrieval evaluation (RetrieverEvaluator)Part 1-2 · hit_rate + mrr over a golden set · from_metric_names
✓Mean Reciprocal Rank (definition + formula)Part 2 · 1/rank of the first hit, averaged
◐RAGAS + retrieval ranking metrics (precision, recall, NDCG)Named here; full depth in b3
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · A "golden set" is...

Golden means the right answer was agreed in advance, so the machine can check against it forever. Without known answers you are back to vibes.

2 · The right answer is retrieved at rank 2. Its contribution to MRR is...

MRR uses 1/rank of the first correct hit. Rank 2 gives 0.5; rank 1 gives 1.0; never found gives 0. Only the first hit counts.

3 · Widening k from 1 to 5 can raise Hit Rate but leaves Precision@1 unchanged because...

Hit Rate@k counts a hit anywhere in the top k, so a bigger k finds more. Precision@1 only cares about position 1, which widening k cannot alter.

Builder session 1 cheat sheet · pin this

The eval loopGolden set → run the system → compare to expected → score. Every eval is this loop.
Golden setQuestions with known-right answers, agreed in advance. Small but representative beats big but skewed.
Hit Rate@kFraction of questions where the right doc is in the top k. Answers "did we find it".
MRRAverage of 1/rank of the first right answer. Answers "how high did it rank". 0 if never found.
Precision@1Right answer ranked first. The number that matters when users see one result. k cannot change it.
Rank over thresholdTrust rank-based metrics over absolute similarity scores, which drift by model.
Cheap firstHit Rate + MRR need only ids and a known-right id - no LLM. The fastest regression signal.
Misses = to-do listEvery miss on the scorecard is a retrieval improvement waiting to happen.