learn-rag-with-phoebe / Builder session 3 of 10
Learn RAG with Phoebe · Builder track · Session 3 of 10

Vector databases: where Recall's chunks live

In b2 Recall learned to cut documents into clean chunks - but those chunks and their vectors were sitting in a Python list, gone the moment the process exits. A real assistant needs a store: something that persists vectors, indexes them so search stays fast as they grow into the millions, and filters by metadata. Today Recall gets that home. We learn Chroma end to end, then meet pgvector and FAISS as the same idea in different houses, and finally look under the hood at the three index types - flat, IVF, and HNSW - that decide the speed-versus-memory trade every vector store makes.

🟡 Builder track Practitioners · real Python Live playground included 45 min
0-3 · Welcome 3-20 · Chroma 20-40 · pgvector, FAISS, indexes 40-45 · Q&A
Part 0

From a Python list to a real store

You can build a working retriever with a list of vectors and a loop that computes cosine similarity - that is exactly what b1 did. It even scales to a few thousand chunks. But the moment you want persistence across restarts, sub-second search over millions of chunks, and the ability to filter by "only board notes from Q3", you need a vector database. The good news: they all wrap the same embed-and-rank idea you already understand. We start with the one that is easiest to run.

Live - presented in session Self-study - full depth after class ★ Try it now Sources covered at the end
★ What you build today Recall's first real store: a Chroma collection you can create, add documents to, and query - with metadata filters - in a handful of lines. Plus the working knowledge to reach for pgvector when you already run Postgres, FAISS when you want raw in-process speed, and the right index type for the scale you are at.
Part 1 · covers the vector-store lifecycle in Chroma

Chroma, end to end 10 min live

Chroma is the friendliest place to start because it runs locally with one pip install, embeds your text for you, and persists to disk automatically. The whole lifecycle is four moves: create a client, create a collection, add documents, query. Learn these four and you understand every vector database - the others just change the words.

1 · Client PersistentClient(path) 2 · Collection get_or_create 3 · add() docs · metadata · ids embeds automatically 4 · query() text · n_results where filter Client → collection → add → query. Every vector store is a dialect of these four moves.
🔍 Click to zoom - the four-move lifecycle of a Chroma store
LiveThe whole store in a dozen lines5 min▶

Here is Recall's store, from empty to answering a filtered query. Note the two client choices: PersistentClient writes to disk so your vectors survive a restart, while chromadb.Client() is in-memory and vanishes when the process ends.

Python · create, add, and query a Chroma collectionimport chromadb # persists to disk automatically; chromadb.Client() is in-memory only client = chromadb.PersistentClient(path="./recall_db") collection = client.get_or_create_collection(name="md_notes") collection.add( ids=["A1", "A2"], documents=["MD committed to a 3-year cloud spend cap.", "MD prefers an aisle seat on all flights."], metadatas=[{"source": "board"}, {"source": "travel"}], ) results = collection.query( query_texts=["cloud spend commitment"], n_results=3, where={"source": "board"}, # metadata filter ) print(results["documents"])

Chroma embeds both your documents and your query for you with a built-in model, so you never call an embedding API by hand here - though you can plug in your own. The ids let you update or delete specific chunks later; the metadatas are what make the where filter possible.

LiveMetadata filters: retrieval with a WHERE clause3 min▶

Real corpora are not flat. You want "the answer, but only from board notes" or "only incidents marked SEV1". Chroma's where filter narrows the search to matching metadata before ranking by similarity - the same instinct as a SQL WHERE, applied to vector search.

  • Operators. Chroma's where supports $eq $ne $gt $gte $lt $lte $and $or $in $nin $contains $not_contains - enough to express most real filters.
  • Filter first, rank second. The filter prunes the candidate set, then similarity ranks what survives. This is faster and more precise than ranking everything and filtering after.
  • Metadata is a design choice. Whatever you might want to filter on later - source, date, author, severity - attach it at add() time. You cannot filter on what you did not store.
This is why b2's chunking carried metadata When you chunked the MD notes, each chunk kept its source tag. That was not decoration - it is what lets Recall answer "what did the board decide?" without dredging up travel notes. Chunk shape and metadata are two halves of the same design.
Self-studyUpdate and delete: a store is not write-once3 min read▶

Corpora change - notes get corrected, incidents get resolved, pages get rewritten. Because every chunk went in with a stable id, Chroma lets you revise the store in place rather than rebuilding it from scratch.

Python · upsert a changed chunk and remove a stale onecollection.upsert( ids=["A1"], documents=["MD raised the cloud spend cap to a 4-year commitment."], metadatas=[{"source": "board"}], ) # re-embeds and replaces the chunk with id A1 collection.delete(ids=["A2"]) # drop a chunk that no longer applies print(collection.count()) # how many chunks are in the store now

This is why id discipline matters from day one. If your chunk ids are derived from something stable - a document id plus a chunk index, say - then re-ingesting an updated document cleanly overwrites its old chunks instead of leaving duplicates behind. Sloppy ids are how stores quietly fill with stale answers.

Part 2 · covers pgvector, FAISS, and the three index types

pgvector and FAISS - same idea, different homes 9 min live

Chroma is one house for your vectors. Two others come up constantly: pgvector, when you already run PostgreSQL and want vectors to live beside your relational data, and FAISS, when you want raw in-process speed and full control. The vectors and the cosine ranking are identical; what changes is where they live and how they are indexed.

Flat exact · linear scan Checks every vector. Minimal memory, no training. IVF train cells · probe a few Searches nearest cells only. Lower memory, must train. HNSW graph · fast · high memory Hops along a graph to neighbors. Best recall + speed, no training. Gold dot = the query. Flat is exact; IVF and HNSW are approximate - they trade a little recall for a lot of speed.
🔍 Click to zoom - Flat, IVF, and HNSW: the three ways to index vectors for search
Self-studypgvector: vectors as a Postgres column4 min read▶

pgvector adds a vector type to PostgreSQL, so your embeddings become just another column - queryable with SQL, joinable with your existing tables, backed up with your existing database. If you already run Postgres, this is often the lowest-friction store you can pick.

SQL · enable pgvector, index with HNSW, query by cosineCREATE EXTENSION vector; CREATE TABLE items ( id bigserial PRIMARY KEY, embedding vector(1536) ); -- build an HNSW index using the cosine opclass CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops); -- <=> is the cosine distance operator SELECT * FROM items ORDER BY embedding <=> '[...]' LIMIT 5;

The operators encode the distance metric: <-> is L2, <=> is cosine, <#> is negative inner product, and <+> is L1. One rule trips everyone up: the index opclass must match the operator you query with - vector_cosine_ops goes with <=>. Mismatch them and Postgres quietly ignores your index and scans the whole table.

Self-studyFAISS: raw speed, in your process4 min read▶

FAISS is a library, not a server - it holds vectors in memory and searches them extremely fast, right inside your Python process. It is the tool when you want maximum control and speed and are willing to manage persistence and metadata yourself.

Python · build a flat FAISS index and search itimport faiss index = faiss.IndexFlatL2(d) # exact search, no training needed index.add(xb) # xb: (n, d) matrix of document vectors D, I = index.search(xq, k) # xq: query vectors, k: neighbors # I = integer ids of the nearest vectors, D = their distances

Swap the index class to change the trade-off: IndexIVFFlat needs .train() on sample data before .add(), and its nprobe setting tunes speed against accuracy; IndexHNSWFlat is the graph index and needs no training. One catch to design around: FAISS returns integer ids only - it stores no documents or metadata, so you keep a side table mapping ids back to your chunks.

LiveChoosing a store, and choosing an index3 min▶

Two decisions, made in order. First, the store:

  • Chroma - starting out, prototyping, or you want batteries-included with automatic embedding and metadata. Recall's choice.
  • pgvector - you already run Postgres and want vectors beside relational data, one backup, one set of ops.
  • FAISS - you want the fastest in-process search and full control, and can manage persistence and metadata yourself.

Then, the index type inside whichever store:

  • Flat - exact, minimal memory, linear scan. Perfect up to tens of thousands of vectors.
  • IVF - lower memory, but you must train it on your data first. Good middle ground at larger scale.
  • HNSW - fastest queries and best recall, no training, but the highest memory use. The common default when scale and latency matter.
Exact vs approximate Flat is exact - it truly checks every vector. IVF and HNSW are approximate-nearest-neighbor (ANN) indexes: they skip most vectors and accept a tiny chance of missing the true best match in exchange for enormous speedups. At millions of vectors that trade is not optional, it is the only way search stays fast.
Live★ Reinforce: a query still returns ranked chunks4 min▶

Whichever store and index you pick, the thing you get back is unchanged from b1: a ranked list of chunks. Run a query against Recall's Corpus A and hold that in mind - Chroma's query(), pgvector's ORDER BY ... LIMIT, and FAISS's search() all produce exactly this shape.

Honesty note - what this playground really does To run with zero network calls, this tool uses a simplified embedding: term frequency over a small vocabulary plus a synonym map, and the same function embeds both the documents and your query. Real embedding models are neural and far richer, and a real vector database would index these vectors with HNSW or IVF rather than scanning them. But the ranking math - cosine similarity over vectors - is exactly what production RAG uses. Only the embedder is simplified; the retrieval mechanic is real.
Build-along · stand up Recall's store

Run Chroma locally and query it ★ 12 min

Time to put Recall's chunks in a real store. This runs on your laptop with one install - no server, no API key.

Install and create. pip install chromadb, then create a PersistentClient(path="./recall_db") and a get_or_create_collection("md_notes").

Add Corpus-A-like notes. Write six short notes of your own - a couple about "board decisions", a couple about "travel", a couple about "people". Give each an id and a {"source": ...} metadata tag, and add() them.

Query with and without a filter. Run the same query twice: once plain, once with where={"source": "board"}. Confirm the filter changes which chunks come back.

Prove persistence. Restart Python, re-open the same client and collection, and query again without re-adding. Your vectors are still there - that is what a store buys you.

Homework

Before session b4 ◐ 40 min total

Source material

Official sources covered

Taught from official docs. This page covers ~80% of their working content on getting started, indexing, and querying - the rest (hosted/cloud tiers, sharding, tuning) stays with the source.

✓Chroma · Docs (getting started)Part 1 · PersistentClient vs in-memory · get_or_create_collection · add · query · where operators
✓pgvector · READMEPart 2 · vector type · HNSW index · distance operators <-> <=> <#> <+> · opclass must match operator
✓FAISS · WikiPart 2 · IndexFlatL2 · IndexIVFFlat train + nprobe · IndexHNSWFlat · integer-id-only results
◐Weaviate · Vector Index (ANN concepts)Part 2 · exact vs approximate · HNSW graph intuition · the recall/speed/memory trade
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · In Chroma, what is the difference between PersistentClient and chromadb.Client()?

PersistentClient(path=...) persists to disk automatically. chromadb.Client() keeps everything in memory only, so it is lost on exit.

2 · In pgvector, which operator ranks by cosine distance - and what must match it?

<=> is the cosine distance operator. The index opclass (vector_cosine_ops) must match it, or Postgres ignores the index.

3 · Which index type is fastest with the best recall, but uses the most memory and needs no training?

HNSW gives the fastest queries and best recall with no training, at the cost of high memory. IVF is lower-memory but must train; Flat is exact but linear.

Builder session 3 cheat sheet · pin this

Why a vector DBPersistence, fast search at millions of vectors, and metadata filtering. Wraps b1's embed-and-rank.
Chroma lifecycleClient → get_or_create_collection → add(docs, metadatas, ids) → query(text, n_results, where).
Persistent vs in-memoryPersistentClient(path) survives restarts; chromadb.Client() is memory-only.
Chroma where operators$eq $ne $gt $gte $lt $lte $and $or $in $nin $contains $not_contains. Filter first, rank second.
pgvectorvector(1536) column; HNSW index; operators <-> L2, <=> cosine, <#> neg inner, <+> L1. Opclass must match.
FAISSIn-process. IndexFlatL2 (exact), IndexIVFFlat (train + nprobe), IndexHNSWFlat (graph). Integer ids only.
Index typesFlat = exact/min memory; IVF = trained/lower memory; HNSW = fastest/best recall/high memory.
Exact vs ANNFlat is exact. IVF and HNSW are approximate - trade a little recall for big speed at scale.
★ Recall's status after b3 Recall now has a real home: a Chroma collection that persists its chunks, indexes them for fast search, and filters by metadata. It is no longer a script that forgets everything on exit - it is a store you can query. In b4 we close the loop: take a retrieved chunk, hand it to a language model, and turn retrieval into an actual answer.