Chunking for retrieval: why your chunk size is why the answers are wrong

01What fixed-size splitting destroys

The default recipe - split every 512 tokens with 50 tokens of overlap - is a text-processing decision applied to a retrieval problem. It cuts tables in half so the header row lands in one chunk and the numbers in another. It separates a heading from the paragraph it governs. It breaks a numbered procedure across a boundary so step four arrives with no indication of what procedure it belongs to.

The resulting chunk is often still readable, which is why the problem survives review. What it has lost is the terms that made it retrievable. A chunk that says 'the limit is 40 requests per second' embeds nowhere near a question about rate limiting on the export API, because the words 'rate limit' and 'export API' were in the heading two chunks earlier.

02Chunk size is a trade, not a setting

Small chunks give precise embeddings because a short passage about one topic produces a vector that is genuinely about that topic. They also fragment context, so an answer spanning three paragraphs requires retrieving three chunks that may not all rank highly. Large chunks preserve context and dilute the embedding - a 2,000-token chunk covering six subtopics has a vector that is near the centroid of all of them and strongly similar to nothing.

There is no universally correct size, but the shape of the trade is consistent, and the right answer depends on your query pattern more than your corpus. Fact-lookup questions want small chunks. Questions requiring synthesis across a section want larger ones. If your traffic contains both, that is an argument for retrieving small and expanding to the parent, not for splitting the difference.

03Split on structure, not on character count

Documents already contain boundaries their authors intended: headings, sections, list items, table rows, function definitions, slide breaks. Splitting along those lines produces chunks that are units of meaning, and it costs less engineering than most teams expect because the structure is already in the markup.

Different content types need different rules and mixing them into one splitter is a common mistake. Code should split on function or class boundaries and keep imports attached. Tables should stay whole or repeat their header row in every fragment. Transcripts should split on speaker turns or topic shifts rather than mid-sentence. A pipeline with three type-aware splitters outperforms one clever universal splitter, reliably.

• Parse to structure first - HTML or Markdown headings, PDF layout, code AST.
• Never split inside a table, code block or list item.
• Fall back to sentence boundaries, then to character count, only when structure runs out.
• Enforce a maximum size after structural splitting, since one section may be enormous.

04Give every chunk its context back

The highest-return change available in most retrieval systems is prepending context to each chunk before embedding: the document title, the heading path, and a sentence describing what this passage is about within the document. A chunk that starts 'Export API / Rate limits: The limit is 40 requests per second' matches queries the bare sentence never could.

Generating that context with a cheap model at ingestion is inexpensive relative to what it buys, and it is done once per document rather than per query. Keep the original text separately for what you pass to the generation model - you want the enriched version for the embedding and the clean version for the answer, because the prepended header is noise in the final prompt.

05Decouple what you retrieve from what you read

The small-to-big pattern resolves the size trade rather than compromising on it. Index small, precise chunks so that matching is accurate, but when a small chunk is retrieved, return its parent section to the generation model. You get the precision of the small embedding and the completeness of the larger passage.

Sentence-window retrieval is the same idea at finer grain: embed individual sentences, return the surrounding few. Both require your store to hold the parent relationship, which is a schema decision to make at ingestion. Retrofitting it means reprocessing the corpus, so decide early even if you start with flat chunks.

06Measure retrieval separately from generation

The diagnostic that saves the most time is separating the two stages. Take a set of real questions, record which chunks should contain the answer, and measure whether retrieval returns them in the top k. If the right chunk is not being retrieved, no amount of prompt engineering downstream will fix the answer, and teams routinely spend weeks tuning the generation step to compensate for a chunking problem.

When you change chunking, re-run that retrieval measurement before looking at end-to-end quality. Chunking changes require full re-embedding of the corpus, which is the most expensive thing in the pipeline to iterate on, so build the evaluation set first and test on a representative sample of documents rather than the whole corpus each time.