learn-rag-with-phoebe / Builder session 6 of 10
Learn RAG with Phoebe · Builder track · Session 6 of 10

Reranking: retrieve wide, then get precise

Your retriever's #1 result is often not the best answer - it is the best answer a single compressed vector could find. An embedding squeezes a whole passage into one point, and that compression loses detail. Reranking fixes it with a two-stage move: a fast retriever casts a wide net, then a slower, sharper model rescores just those candidates so the truly best chunk rises to the top. This session covers bi-encoders vs cross-encoders, the retrieve-wide-then-rerank-narrow funnel, MMR for trimming near-duplicates, and query expansion / HyDE for catching what the first pass missed. This is where Recall goes from useful to precise.

🟠 Builder track Practitioners · some Python Concept-led session Session 6 of 10
0-3 · Recap 3-22 · Two-stage rerank 22-40 · MMR + HyDE 40-45 · Q&A
Part 0

Why the top result is not always the best result

By b5 Recall retrieves by meaning, filters by metadata, and fuses dense with sparse. But every one of those still ranks by comparing precomputed vectors, and a vector is a lossy summary. Two passages can look equally close to a query in embedding space when a careful reader would rank one far above the other. Reranking is that careful reader - a model that looks at the query and a candidate together and scores their real relevance. The catch is it is slow, so you only ever run it on a shortlist. That constraint shapes the whole design.

Live - presented in session Self-study - full depth after class ★ Try it now Sources covered at the end
★ What you build today A clear model of bi-encoder vs cross-encoder scoring, the two-stage retrieve-wide-then-rerank-narrow funnel and why latency forces it, plus two recall-boosting tricks - MMR to kill near-duplicates and query expansion / HyDE to catch chunks the first pass missed.
Part 1 · covers bi-encoder vs cross-encoder + the two-stage funnel

Bi-encoder vs cross-encoder, and the two-stage funnel 9 min live

The retriever you have used all track is a bi-encoder: it encodes the query and each document separately into vectors, then compares them with cosine. That is why it is fast - document vectors are precomputed once and reused forever. A cross-encoder is different: it feeds the query and one document together through a single transformer and outputs a query-specific relevance score. Far more accurate, and far too slow to run over a whole corpus. So you stage them.

query + whole corpus stage 1 bi-encoder fast · vector search <100ms wide net: top ~150 candidates stage 2 cross-encoder slow · scores each query+doc pair top ~20 reordered to the generator Cheap where the corpus is huge, expensive only on the shortlist. That funnel is the whole trick.
🔍 Click to zoom - retrieve wide with a fast bi-encoder, rerank narrow with a slow cross-encoder
LiveBi-encoder vs cross-encoder, side by side3 min▶

The difference is when the query and document meet:

  • Bi-encoder (your retriever). Query and document are encoded into vectors independently. Document vectors are computed once at index time, so a query is just "embed the query, cosine against the precomputed vectors". Fast and scalable - but the two texts never interact, so subtle relevance is lost.
  • Cross-encoder (the reranker). The query and one document are concatenated and passed through the transformer together, so attention can compare them token by token. The output is a single relevance score for that exact pair. Much more accurate - but nothing is precomputable, so you must run a fresh forward pass for every candidate.
Why the bi-encoder loses precision Encoding a passage into one vector compresses its full meaning into a fixed list of numbers - detail is thrown away, which costs you recall and precision at the margin. The cross-encoder never compresses; it reads the query and the passage in the same context. That is the entire quality gap, and the entire reason it is slow.
LiveWhy you rerank the shortlist, not the corpus3 min▶

The whole two-stage design exists because of latency. A bi-encoder vector search over millions of chunks returns in well under 100ms because the heavy work happened at index time. A cross-encoder has no precomputed anything - it runs a full model pass per query+document pair.

  • The numbers force it. Reranking on the order of 40 million records with a cross-encoder on a V100 GPU would take more than 50 hours for a single query. You cannot rerank a corpus. You can rerank 150 candidates in a blink.
  • So you funnel. Stage 1 (bi-encoder) casts a deliberately wide net - Anthropic's contextual-retrieval work retrieves the top ~150 chunks - precisely so the right answer is somewhere in the candidate set even if it is not #1. Stage 2 (cross-encoder) rescores those ~150 and keeps the top ~20 for the generator.
  • Recall then precision. Stage 1 optimizes recall (get the right chunk into the net); stage 2 optimizes precision (put it at the top). Neither model could do both jobs affordably alone.
Real world

Wider net, then sharper sort. A team (anonymized) was retrieving k=5 straight from the bi-encoder and complained the right doc was "usually there but rarely first". Rather than tune the embedder, they retrieved 100 candidates and added a cross-encoder rerank down to 5. Same embeddings, same corpus - the correct chunk moved from a scattered rank into the top two on most queries. The fix was staging, not a better model.

Self-studyAlso: stuffing too many chunks hurts the LLM2 min▶

There is a second reason reranking matters: you cannot just crank k up to be safe. Handing the generator 50 loosely relevant chunks does not raise accuracy - it lowers it. The model spreads attention across noise, is more likely to cite the wrong passage, and every extra chunk costs tokens and latency. Reranking lets you retrieve wide (high recall) yet feed the LLM a small, high-precision set - the best of both instead of a bad trade.

The shape to remember Retrieve wide so the answer is in the net; rerank narrow so the LLM sees only the best. High recall at stage 1, high precision at stage 2, a short final context for the generator. Reranking is what lets those goals coexist.
Part 2 · covers MMR diversity + query expansion / HyDE

MMR diversity and query expansion / HyDE 7 min live

Two more levers shape what reaches the generator. Query expansion / HyDE works at the input side: widen or rewrite the query so stage 1 retrieves chunks the raw question would have missed. MMR works at the output side: trim near-duplicate results so your final set is diverse rather than five phrasings of the same fact. One improves recall, the other improves the usefulness of the top-k.

raw query expand / HyDE + related terms, a hypothetical answer wide retrieve (candidates) refund window · 5-7 days refunds take 5-7 days how long refunds take return must be unused MMR drop near- duplicates diverse top-k Expand to find more; MMR to keep the final few varied instead of five ways of saying one thing.
🔍 Click to zoom - expand the query going in, trim duplicates with MMR coming out
LiveQuery expansion and HyDE3 min▶

A short or oddly worded query retrieves poorly - the b1 lesson that vague queries are where retrieval is weakest. Query expansion fixes it before retrieval runs, and the DeepLearning.AI Chroma course teaches two flavours:

  • Expansion with related terms. Use an LLM to add synonyms and related phrasing to the query, so the embedding lands nearer more of the relevant chunks. "refund?" becomes "refund, return policy, money back, days to process" - a richer vector, a wider net.
  • HyDE (hypothetical document embeddings). Ask an LLM to write a plausible answer to the question, then embed that hypothetical answer instead of (or alongside) the question. Answers look like documents, so an answer-shaped vector matches real answer passages better than a question-shaped one. It directly attacks the b1 problem that questions rarely share words with their answers.
The cost of expansion Both tricks add an LLM call before retrieval, so they add latency and can drift off-topic if the hypothetical answer hallucinates. Use them where recall is the bottleneck - short queries, sparse corpora - not everywhere by default. Measure hit rate (b4) with and without.
LiveMMR: maximal marginal relevance2 min▶

A wide retrieve often returns near-duplicates: three chunks that all state the same fact in slightly different words. Feeding all three to the generator wastes context on redundancy and can crowd out a different, also-relevant chunk. MMR (maximal marginal relevance) picks results that are relevant to the query and different from what it has already picked - each new chunk must earn its slot by adding something new.

  • What it does. Instead of taking the top-k by pure similarity, MMR balances relevance to the query against dissimilarity to the already-selected results, trimming redundant near-duplicates.
  • When to reach for it. Broad questions ("summarize what went wrong across incidents") where you want coverage of distinct points, not the single best point repeated. Most frameworks expose it as a search type - LangChain has a search_type="mmr" retriever option.
Live★ See it: top-k now, imagine the rerank4 min▶

This is the bi-encoder stage over Corpus B - the raw, un-reranked top-k you have used since b4. Run a broad query and read the list as a candidate set, not a final answer: is the truly most relevant incident at #1, or merely in the net? Are there near-duplicates a cross-encoder would reorder and MMR would thin?

Honesty note - what this playground really does This is only stage 1: a simplified lexical bi-encoder (term frequency + synonyms, offline) ranking by cosine, exactly as in earlier sessions. There is no real cross-encoder here - reranking, MMR, and HyDE are concepts to picture over this candidate set, not code the playground runs. The cosine ranking you see is the real stage-1 mechanic; the stage-2 sharpening is what you would add in production.

Pick a query where #1 looks arguable, then describe in one line how a cross-encoder reading each query+incident pair together would likely reorder it, and which near-duplicate MMR would drop.

Build-along · take it further

Reorder a result set on paper ★ 10 min · the Corpus B playground

You do not need a cross-encoder loaded to reason like one. Use the Corpus B playground above and practise the two-stage instinct by hand.

Find a near-duplicate pair. Run a broad query (e.g. "payment problems") at k=5 and look for two results that say almost the same thing - two payment-outage incidents, say. Note their ranks.

Rerank on paper. Reading each query+incident pair carefully (as a cross-encoder would), decide which of the two is genuinely more relevant to the exact question, and write the order you would put the top-5 in. Note where it differs from the cosine order.

Apply MMR. From your reranked list, drop the near-duplicate that adds the least new information and pull up the next distinct incident. That is MMR by hand - relevance plus diversity.

Expand the query. Take a short query that retrieved poorly and rewrite it with related terms, or sketch a one-sentence hypothetical answer (HyDE). Say which chunks that richer query would now reach that the terse one missed.

★ Recall's status after b6 Recall is now precise: it retrieves a wide candidate set, understands how a cross-encoder rerank would reorder it, trims near-duplicates with MMR, and can widen thin queries with expansion or HyDE. The right chunk not only makes the net - it lands at the top. In b7 we stop trusting Recall blindly: we make it ground every claim in its retrieved chunks, cite sources, and refuse when the best chunk simply is not good enough.
Homework

Before session b7 ◐ 40 min total

Source material

Official sources covered

Taught from official docs and a vendor course. This page covers ~80% of their working content on reranking and query-side recall tricks - the model-training internals and hosted-reranker billing stay with the source.

✓Pinecone · Rerankers / two-stage retrievalPart 1 · bi-encoder vs cross-encoder · retrieve wide then rerank narrow · latency budget
✓DeepLearning.AI · Advanced Retrieval for AI with ChromaPart 2 · query expansion · HyDE (hypothetical answer) · cross-encoder rerank
◐Anthropic · Contextual Retrieval (reranking funnel)Part 1 · top ~150 candidates → rerank → keep top ~20
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · How does a cross-encoder differ from a bi-encoder?

A bi-encoder encodes query and doc separately (fast, precomputable). A cross-encoder reads the pair together for an accurate, query-specific score - too slow to run over a whole corpus.

2 · Why do you rerank only a candidate set (e.g. ~150) rather than the whole corpus?

Cross-encoders have nothing precomputed - reranking ~40M records on a V100 would take 50+ hours. Retrieve wide fast, then rerank the shortlist.

3 · What problem does MMR (maximal marginal relevance) address?

MMR trims redundant near-duplicates by favouring results that are both relevant to the query and different from those already selected.

Builder session 6 cheat sheet · pin this

Bi-encoderEncodes query + doc separately; cosine over precomputed vectors. Fast, your retriever.
Cross-encoderFeeds query+doc together through one transformer. Accurate, query-specific, slow.
Two-stage funnelRetrieve wide (fast bi-encoder) → rerank narrow (slow cross-encoder). ~150 → ~20.
Why stagedCross-encoder on a full corpus = 50+ hrs. Only rerank the candidate set.
Recall then precisionStage 1 gets the answer in the net; stage 2 puts it at the top.
Too many chunks hurtStuffing k high adds noise + tokens; the LLM cites the wrong passage.
Query expansion / HyDEAdd related terms, or embed an LLM-written hypothetical answer, to widen retrieval.
MMRMaximal marginal relevance - trims near-duplicates; relevance + diversity.