learn-rag-with-phoebe / Builder session 4 of 10
Learn RAG with Phoebe · Builder track · Session 4 of 10

Retrieval basics: ask, rank, take the top few

You have embeddings (b1), chunks (b2), and a vector database to hold them (b3). Now the payoff: given a question, pull the passages most likely to answer it. That is retrieval, and at its core it is one move - embed the query, score every chunk by cosine, sort, keep the top k. This session makes that move concrete in Python, wraps it behind the retriever interface every framework exposes, and teaches you the skill that separates people who ship RAG from people who ship broken RAG: reading a ranked result set critically. Recall now answers real questions over Corpus A.

🟡 Builder track Practitioners · some Python Live playground included Session 4 of 10
0-3 · Recap 3-18 · top-k search 18-40 · Build-along 40-45 · Q&A
Part 0

From a store of vectors to an answer

In b3 you loaded Recall's notes into a vector store. A store you never query is just an expensive filing cabinet. Retrieval is the query: it takes a natural-language question and returns the handful of chunks most likely to contain the answer, ordered best-first. Everything downstream - the prompt you build, the citations you show, the answer the model writes - is only as good as this ranked list. So we learn to build it, and to read it with a suspicious eye.

Live - presented in session Self-study - full depth after class ★ Try it now Sources covered at the end
★ What you build today A working retrieve(query, k) function over Recall's Corpus A, the same call expressed through a real framework's retriever interface, and a repeatable habit for judging whether a result set is trustworthy - including a hand-computed hit rate you will lean on again in b9.
Part 1 · covers top-k similarity search

Top-k similarity search, end to end 7 min live

Retrieval is four steps that never change: embed the query with the same model you used for the documents, score the query vector against every stored chunk with cosine similarity, sort those scores high to low, and take the top k. The vector database does the middle two fast; you supply the query and choose k.

query vector "cloud spend cut?" cosine vs every chunk A1 · 0.71 A7 · 0.58 A3 · 0.34 A2 · 0.11 A5 · 0.04 sort, then cut at k=3 #1 A1 0.71 #2 A7 0.58 #3 A3 0.34 cut line k cut to the generator Chunks below the k cut are discarded for this query - even if they were relevant. That is the risk k controls.
🔍 Click to zoom - embed, score, sort, take top-k: the whole retrieval move
Self-studyThe retriever in a dozen lines of Python3 min▶

Written by hand, retrieval is exactly the four steps. Here it is over Corpus A - embed the query, cosine against every chunk, sort, slice:

Python · a top-k retriever from scratchimport numpy as np from openai import OpenAI client = OpenAI() def embed(texts): resp = client.embeddings.create(model="text-embedding-3-small", input=texts) return [np.array(d.embedding) for d in resp.data] def cosine(a, b): return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b))) # Corpus A - the MD's chief-of-staff notes docs = [ "In the Q2 board call the MD committed to cutting cloud spend by 15 percent.", "The MD prefers aisle seats and never books red-eye flights.", "Expenses over 500 dollars need a receipt photo and a business reason.", ] doc_vecs = embed(docs) def retrieve(query, k=4): q = embed([query])[0] scored = [(cosine(q, v), doc) for v, doc in zip(doc_vecs, docs)] scored.sort(reverse=True) # highest cosine first return scored[:k] # take the top k for score, doc in retrieve("what did the MD promise about cloud spend?"): print(f"{score:.3f} {doc}")

In production you never loop in Python - the vector store scores all chunks with an ANN index (b3). But the contract is identical: query in, ranked chunks out. If you can read this function, you can read any retriever.

Self-studyThe same call against a real vector store2 min▶

Every vector database ships a one-line version of the loop above. With Chroma you hand it the query text and how many results you want; it embeds and ranks for you:

Python · Chroma top-k queryimport chromadb client = chromadb.Client() collection = client.get_or_create_collection("recall_corpus_a") # ... documents added in b3 ... results = collection.query( query_texts=["what did the MD promise about cloud spend?"], n_results=4, # this is your k ) for doc, dist in zip(results["documents"][0], results["distances"][0]): print(f"{dist:.3f} {doc}") # distance: smaller = closer

Note n_results is k, and Chroma returns a distance (smaller = more similar) rather than a cosine score (larger = more similar) - same ranking, flipped sign. Read your store's docs so you know which way "good" points.

LiveChoosing k, and rank vs threshold3 min▶

k is the one knob you own on day one, and it trades off two failures:

  • k too small - you miss relevant chunks that sat just below the cut. If the answer lived at #4 and k=3, the model never sees it. This is a silent failure: the answer looks confident and is simply incomplete.
  • k too big - you drag in loosely related noise. That costs tokens (every chunk goes into the prompt), costs latency, and can distract the model into citing the wrong passage. More context is not more accuracy.

A common starting point is k=4 and you tune from there against real questions. The second habit:

Rank over absolute threshold Do not hard-code "keep everything above cosine 0.8". Absolute scores drift by embedding model, by corpus, even by query length, so a fixed cutoff that works today breaks when you swap models. Trust the ordering - return the top k - and save the "is the best score good enough to answer at all?" question for the refusal logic in b7.
Part 2 · covers the retriever interface + reading results

The retriever as a component, and how to read its output 6 min live

Frameworks wrap that four-step move in a reusable object called a retriever: a component with one job - take a query string, hand back a ranked list of documents. Because the interface is fixed, you can swap what is behind it (a different store, hybrid search in b5, a reranker in b6) without touching the code that consumes the results. The generator just asks the retriever for chunks.

query retriever embed query score + sort in the store return top-k docs ranked docs generator (the LLM) swap the internals - dense, hybrid, reranked - the interface never changes One stable contract: query in, ranked documents out. That is why retrieval is pluggable.
🔍 Click to zoom - the retriever is a swappable component feeding the generator
Self-studyThe LangChain retriever interface2 min▶

In LangChain the vector store gives you two front doors. The direct one is similarity_search; the composable one is .as_retriever(), which returns an object any chain can call:

Python · LangChain similarity_search and .as_retrieverfrom langchain_openai import OpenAIEmbeddings from langchain_chroma import Chroma vector_store = Chroma( collection_name="recall_corpus_a", embedding_function=OpenAIEmbeddings(model="text-embedding-3-small"), ) # direct call - returns the top 4 Documents, ranked docs = vector_store.similarity_search("what did the MD promise about cloud spend?", k=4) for d in docs: print(d.page_content) # the same store as a reusable retriever component retriever = vector_store.as_retriever(search_kwargs={"k": 4}) docs = retriever.invoke("travel seat preference")

Both do the identical four steps. similarity_search(query, k=4) is what you reach for while exploring; the retriever is what you wire into a chain so the rest of the pipeline never needs to know how retrieval happens.

LiveReading a ranked result set critically4 min▶

The habit that saves you weeks: never trust a retriever you have not eyeballed. For any query, ask three things of the ranked list:

  • Is #1 actually relevant? Read the top chunk. Does it genuinely address the question, or did it just share some vocabulary? A high score on the wrong chunk is worse than a low score on nothing.
  • Is the right chunk anywhere in the top-k? Even if it is not #1, being at #2 or #3 is fine - the generator sees all k. Being absent is the failure that matters.
  • How far did relevance fall off? If #1 is strong and #2-#4 are noise, a smaller k would be cleaner. If several are all plausible, you may be under-retrieving.
Real world

The demo that lied. A team (anonymized) shipped a support bot that looked great in the demo - every question returned a confident answer. When they finally read the top-k for 20 real tickets, the correct article was missing from the top-3 on nearly half of them; the model was fluently paraphrasing the wrong chunk. Nobody had looked at the ranked list. The fix was not a better model - it was measuring hit rate first.

That "is the right chunk in the top-k?" question, counted across many queries, is hit rate - the fraction of queries where the correct document lands in the top-k. It is the cheapest retrieval metric you have, and you can compute it by hand today.

Live★ Build-along: vary the query, watch the ranking move6 min▶

Here is Recall answering over Corpus A with k=4. Type a question, hit Search, and read the ranked list the way Part 2 taught: is #1 right, is the correct note in the top-4, where does relevance drop off? Change a word and watch the cosine scores and the order shift.

Honesty note - what this playground really does To run with zero network calls, this tool uses a simplified lexical embedding: term frequency over a small vocabulary with a synonym map, computed the same way for notes and queries. A real embedding model is neural and far richer. But the mechanic you are grading - cosine ranking, take the top-k - is exactly what production retrieval does. You are reading real ranked output from a toy embedder.

Try "what did the MD promise about cloud spend?" (should surface the Q2 board note), then "aisle or window?" (travel prefs), then something Corpus A cannot answer like "what is our AWS bill?" - and notice it still returns its best guess. Judging when a best guess is too weak to use is session b7.

Build-along · take it further

Compute a hit rate by hand ★ 10 min · the playground above

You do not need a framework to measure retrieval quality - you need five questions and honest eyes. Do this over the Corpus A playground and you will have your first evaluation number.

Write five questions you know Corpus A can answer, and for each note down which note (A1-A8) is the correct source. Example: "when does the daughter graduate?" → A8.

Run each query with k set to 4 (or mentally take the top-3). For each, mark a ✓ if the correct note appears in the top-3, an ✗ if it does not.

Compute hit rate = (number of ✓) / 5. Four out of five is 0.80. That single number is your baseline; every change you make later (chunking, hybrid, reranking) should move it up.

Inspect the misses. For any ✗, read what did rank first. Was it a synonym collision? A too-short query? Write one sentence on why it missed - those sentences become your test cases in b9.

★ Recall's status after b4 Recall can now take a question and return the notes most likely to answer it, ranked best-first - and you can measure how often it gets the right note into the top-k. It retrieves. But it only searches by meaning: ask "which SEV1 incidents happened in March?" and pure similarity flounders. In b5 we give Recall metadata filters and hybrid search over Corpus B's incident tickets.
Homework

Before session b5 ◐ 40 min total

Source material

Official sources covered

Taught from official framework docs. This page covers ~80% of their working content on basic retrieval and the retriever interface - the hosted-infra and advanced-config parts stay with the source.

✓LangChain · RAG tutorial + Retrievers conceptPart 1-2 · similarity_search(query, k=4) · .as_retriever() · the retriever as a component
◐LlamaIndex · Querying / query engine guidePart 2 · the same query-in, ranked-nodes-out interface under different names
✓Chroma · query() / n_resultsPart 1 · collection.query(query_texts, n_results=k) · distance vs cosine
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · What does "top-k" retrieval return?

Top-k means score every chunk by cosine, sort high to low, and return the k highest - a ranking, not a fixed threshold.

2 · You set k too small. What is the likely failure?

Small k misses relevant chunks that sat below the cut; large k drags in noise and cost. k=4 is a common starting point you tune.

3 · Hit rate for retrieval is...

Hit rate = fraction of queries whose correct document lands in the top-k. It is the cheapest retrieval metric and you can count it by hand.

Builder session 4 cheat sheet · pin this

The retrieval moveembed(query) → cosine vs every chunk → sort → take top-k. Never changes.
Top-kReturn the k highest-cosine passages. k is the knob you own on day one.
k trade-offSmall k = miss relevant chunks. Big k = noise + tokens + latency. Start at k=4.
Rank, not thresholdTrust the ordering. Absolute scores drift by model - do not hard-code a cutoff.
Retriever interfaceA component: query in, ranked docs out. Swap the internals freely.
LangChainvector_store.similarity_search(query, k=4) or .as_retriever().invoke(query).
Chromacollection.query(query_texts=[q], n_results=k). Returns distance (smaller = closer).
Hit rateFraction of queries where the correct doc is in top-k. Your first eval number.