The most under-rated decision in RAG
Teams spend weeks choosing an embedding model and picking a vector database, then wonder why retrieval is mediocre. Nine times out of ten the real culprit is chunking. Retrieval can only ever return a chunk you created - so if your chunks are shaped wrong, no model and no database can save you. Good news: chunking is cheap to change and you can feel the difference in minutes. Today we make Recall's chunks deliberate.
Why chunk at all 8 min live
Chunking is the step between "here is a document" and "here are vectors in a database". You break the document into pieces, embed each piece separately, and store each as its own retrievable unit. When a query comes in, you rank the chunks, not the documents. So the chunk is the atom of retrieval - the smallest thing the system can hand back.
LiveThree reasons you cannot skip chunking4 min▶
Chunking is not a nicety - it is forced on you by how embeddings and language models work. Three pressures push in the same direction:
- Embedding models have a size limit. Every embedding model has a maximum input length (a context window). Feed it more text than that and it truncates or errors. A long document simply will not fit as one call.
- One vector per whole document loses detail. Squeeze a 20-page report into a single vector and you get an average of everything it says - a smear. The specific sentence that answers the question is drowned out by the other nineteen pages.
- Retrieval returns the chunk, not the document. You want to hand the language model the precise passage that answers the question, not the entire file. Small, focused chunks make precise retrieval possible and keep the prompt cheap.
LiveThe Goldilocks problem: too big vs too small4 min▶
Chunking is a balancing act, and both ends fail in their own way:
- Too big dilutes meaning. A giant chunk covers many topics, so its single vector points at the average of all of them. It matches lots of queries weakly and none of them strongly - and it stuffs your prompt with irrelevant text.
- Too small loses context. A one-line chunk may be perfectly on-topic but missing the surrounding sentences that make it usable. "It was rejected" retrieves nothing useful without knowing what "it" is.
The paragraph that lost its subject. A team (anonymized) chunked a policy handbook by sentence for maximum precision. Retrieval kept surfacing "This must be approved by two directors." - true, but useless, because the sentence naming what must be approved lived in the previous chunk. Widening to paragraph-level chunks with overlap fixed it instantly.
The craft of chunking is finding the size where each piece is focused enough to rank sharply but complete enough to stand on its own. That is what the strategies in Part 2 are all reaching for.
Self-studyWhat makes a good chunk3 min read▶
Before you learn the strategies, it helps to know what they are all aiming at. A good chunk has three properties you can check by eye:
- One idea. A reader should be able to say what the chunk is about in a single phrase. If it takes two, the chunk is probably two chunks.
- Self-contained. It makes sense on its own, without the sentence before or after it. Pronouns with no antecedent ("it", "this", "they") are the warning sign of a chunk that was cut too tight.
- Retrievable in isolation. If you imagine the question this chunk answers, the chunk should contain enough of the question's language or meaning to be found. A chunk nobody can retrieve is dead weight in your store.
Notice that these pull against each other - "one idea" wants small, "self-contained" wants big. Every strategy in Part 2 is a different way to resolve that tension, and the contextual trick at the end is a way to cheat it.
The strategies 9 min live
There is a ladder of chunking strategies, from dead-simple to clever. You climb it only when the simple rung stops working - most systems live happily near the bottom. Here is the ladder, cheapest and most common first.
| Strategy | How it works | When to use |
|---|---|---|
| Fixed-size | Cut every N characters or tokens, usually with a little overlap. No awareness of structure. | Your default and starting point. Fast, predictable, good enough for most corpora. |
| Recursive (character) | Try to split on big separators first (paragraphs), fall back to smaller ones (lines, spaces) to keep pieces near the target size without breaking mid-thought. | When documents have natural structure - prose, docs, articles. The practical everyday choice. |
| Semantic | Embed sentences and cut where the meaning shifts, so each chunk is one coherent topic. | When topics change unpredictably within a document and fixed sizes cut across ideas. |
| Sentence-window | Index single sentences for precise matching, then expand to neighboring sentences before sending to the LLM. | When you need pinpoint retrieval but the LLM still needs surrounding context to answer. |
| Parent-document | Index small child chunks for precise retrieval, but feed the larger parent chunk to the LLM once a child matches. | When precise matching and rich context pull in opposite directions - a common, powerful pattern. |
| Contextual | Prepend a short generated context (what section, what document) to each chunk before embedding. | When chunks lose meaning out of context - Anthropic's fix for a large class of retrieval misses. |
Self-studyRecursive splitting in Python, the everyday default4 min read▶
The workhorse in practice is LangChain's RecursiveCharacterTextSplitter. It tries a list of separators in order - ["\n\n", "\n", " ", ""] - splitting on paragraph breaks first, then line breaks, then spaces, then raw characters, so it keeps pieces near your target size without cutting across a paragraph if it can help it.
Those numbers - chunk_size=1000, chunk_overlap=200 - are LangChain's defaults and a perfectly good place to start. Change them only once you have a reason, which brings us to the two levers.
Self-studySemantic chunking: cutting where the meaning turns2 min read▶
Fixed and recursive splitters cut by shape - characters and separators. Semantic chunking cuts by meaning: it embeds each sentence, walks through the document, and starts a new chunk wherever the topic shifts (a large jump in embedding distance between consecutive sentences). Each chunk ends up being one coherent thought, whatever its length.
- Upside. No idea gets cut in half by an arbitrary character count, and chunks match their topic tightly.
- Cost. You embed every sentence just to decide the boundaries, so it is slower and pricier to build the index. Reach for it only when fixed and recursive visibly cut across ideas in your corpus.
In practice most teams ship on recursive and keep semantic chunking in their back pocket for the documents that need it.
LiveThe two levers: size and overlap3 min▶
You will spend most of your chunking life tuning exactly two dials:
- Chunk size. Test a range from small (128-256 tokens) to large (512-1024 tokens). Smaller chunks retrieve more precisely; larger chunks carry more context. The hard ceiling is your embedding model's context window - a chunk can never be bigger than what the model can read.
- Overlap. Letting neighboring chunks share some text (the default 200 characters) means a sentence sitting on a boundary still appears whole in at least one chunk. It is cheap insurance against cutting an idea in half.
LiveRetrieve small, feed big: sentence-window and parent-document3 min▶
Two strategies deserve a closer look because they resolve the size tension with a clever split: what you index for retrieval does not have to be what you send to the language model.
- Sentence-window. Index each sentence on its own for pinpoint matching, but when a sentence matches, expand outward to its neighbors before building the prompt. You get precise retrieval and readable context.
- Parent-document. Index small child chunks so matching is precise, but store which larger parent chunk each child came from. On a hit, feed the LLM the parent, not the tiny child. This is a common, powerful production pattern - precise where it counts, contextual where it matters.
Precise hit, useless answer. A support bot (anonymized) matched a one-line troubleshooting step perfectly but answered incompletely, because the step assumed three lines of setup above it. Switching to parent-document retrieval - match the line, send the whole procedure - turned partial answers into complete ones without touching the embedding model.
Feel a well-formed chunk retrieve ★ 10 min · live playground
Recall's first real corpus is Corpus A - the MD's chief-of-staff notes. Each note here is short and about one thing: it is already a clean, well-formed chunk. Type a question and watch a single tidy chunk rise to the top. This is what good chunking earns you - retrieval that returns one focused, usable piece.
Notice how each note behaves like a good chunk should: focused enough to rank sharply, complete enough to make sense on its own. Now imagine one note glued to five others - the query would match the blob weakly and you would hand the LLM a wall of mostly-irrelevant text. That is the whole argument for chunking, felt rather than told.
Chunk your own document three ways ◐ 12 min
Nothing teaches chunking like breaking a document you know well. Pick a real document of your own - a README, a policy, a set of notes.
Cut it three ways. Run RecursiveCharacterTextSplitter at chunk_size=256, then 512, then 1000, each with 20% overlap. Print the chunk count and the first two chunks for each.
Read the seams. For each size, find one chunk that got cut mid-idea and one that stands cleanly on its own. Which size had more clean chunks?
Query in your head. Write one question your document answers. Which size would retrieve the best single chunk to answer it - focused but complete?
Decide. In one line, pick the size you would ship for this document and say why. That judgment, repeated across corpora, is chunking expertise.
Before session b3 ◐ 40 min total
- Take the best chunk size from the exercise and re-run it with three overlap values (0, 100, 200 characters). Confirm that some overlap keeps boundary sentences intact and that too much just wastes storage.
- Implement a tiny contextual-chunking prototype: for three chunks, prepend a one-line context ("From the Q3 board notes, on cloud spend:") by hand, then eyeball how much clearer each chunk reads out of context.
- Read: Pinecone's "Chunking Strategies" guide end to end, and skim the Anthropic Contextual Retrieval post - note the reported drop in retrieval failures.
- Keep your three chunked versions. In b3 you will load one of them into a real vector database.
Official sources covered
Taught from official docs and courses. This page covers ~80% of their working content on chunking - the rest (hosted parsing infra, paid pipelines) stays with the source.
Three questions before you go 🎯 ◐ 90 seconds
1 · Why can't you just embed a whole long document as one vector?
Embedding models have a context limit, and one vector for a long document smears every topic together - the specific answering sentence gets drowned out.
2 · What is the recommended starting point for chunking?
Fixed-size is Pinecone's recommended start. You climb to fancier strategies only when the simple rung stops working.
3 · What does Anthropic's contextual chunking do?
Contextual Retrieval prepends which-document, which-section context to each chunk before embedding, so the chunk carries the context it would otherwise lose.