Chunking is the part of RAG nobody tunes
Chunking determines what retrieval can possibly return. Fixed-size splits cut sentences and separate claims from their context; splitting on document structure with a little overlap, and prepending the section heading to each chunk, consistently outperforms the default character-count approach.
Chunking is the least discussed step in RAG and one of the most decisive. Retrieval can only ever return a chunk, so how you cut the text sets a hard ceiling on what any amount of downstream cleverness can find.
Why fixed-size splitting disappoints
The default in most tutorials is to cut every N characters with some overlap. It is easy and it is indifferent to meaning.
A cut lands mid-sentence. A table is separated from the paragraph explaining it. A qualifying clause — "this applies only to annual plans" — ends up in a different chunk from the rule it qualifies, so retrieval can return the rule without the exception.
The orphaned chunk problem
A chunk reading "It must be enabled before the first sync" is useless in isolation. What must? The document knew; the chunk does not.
Prepending the document title and the current heading to every chunk fixes this almost entirely, and it improves embedding quality at the same time, because the topic is now present in the text being embedded.
Split on structure instead
Most content already tells you where it divides — headings, sections, list items, function definitions. Using those boundaries produces chunks that are coherent units of meaning rather than arbitrary slices of characters.
Cap the size for the occasional enormous section, and merge trivially short ones with their neighbour. Otherwise let the document's own shape decide.
Match the strategy to the content
Prose splits well on paragraphs and headings. Code splits on function and class boundaries; a chunk that ends mid-function retrieves badly. Conversation logs split on turns. Reference tables often should not be split at all.
One splitter applied to every content type is a compromise nobody chose deliberately, and it usually shows up as one document type that mysteriously never retrieves well.
Measure it directly
Chunking is easy to evaluate in isolation, which is what makes it worth tuning first. Take questions with known answers, check whether the chunk containing the answer is retrievable at all, and iterate on the splitter.
That loop is fast, and the gains are usually larger than a reranker will give you.
Frequently asked questions
What chunk size should I use?
Size is the wrong primary question. Split on structure first and let size fall where it does, with a cap. A section of 200 words and one of 600 are both fine; a 400-character slice that begins mid-sentence is not.
Does overlap not waste storage?
It does, and storage is the cheapest thing in the pipeline. A ten to fifteen percent overlap costs very little and prevents the failure where the answer sits exactly on a boundary.
