A reasonable starting point for most enterprise RAG systems is chunks of about 300 to 800 tokens with an overlap of roughly 10 to 20 percent of the chunk size, though the right numbers depend on document type and the embedding model's context window. Smaller chunks, in the 200 to 400 token range, tend to improve retrieval precision because each vector represents a narrower, more specific idea, which helps when questions target a single fact like a number or a clause; larger chunks, up to 800 or 1000 tokens, preserve more surrounding context and work better for questions that need a fuller explanation or span several sentences. Overlap prevents a sentence or idea from being split across two chunks and losing meaning at the boundary, but too much overlap wastes index space and can return near-duplicate results in the top-k. Dense technical documents like contracts or engineering specs often benefit from smaller chunks with structure-aware splitting on clauses or sections, while narrative content like reports or transcripts tolerates larger chunks. These settings should be tuned against a labeled evaluation set rather than fixed in advance. Nanobase AI tests chunk size and overlap empirically against retrieval metrics for each customer's document set before finalizing the pipeline.
Why a copied default number underperforms
Guides frequently state a chunk size range as if it were universal, but that range is a reasonable starting point, not a tuned answer for a specific corpus. The right chunk size depends on how densely a document type packs meaning per sentence, how the embedding model was trained, and how the language model consuming the chunks handles multiple retrieved passages. Copying a number from a tutorial and never revisiting it is one of the most common reasons a RAG system plateaus below its achievable accuracy.
A chunk size copied from a tutorial is a starting hypothesis, not a finished configuration.
What each parameter actually trades off
| Parameter | Increasing it | Decreasing it |
|---|---|---|
| Chunk size | More context per chunk, but each vector represents a blurrier, less specific idea | Sharper, more specific vectors, but answers spanning multiple sentences may get split |
| Overlap | Less chance a boundary cuts through an important sentence, more storage and embedding cost | Lower cost, but higher risk that a key sentence sits only in one chunk near a boundary |
| Number of chunks retrieved (top-k) | Higher chance the right chunk is included, but more noise for the model to filter, and higher token cost | Cleaner context, but higher chance the right chunk is missed entirely |
These three parameters interact, so tuning chunk size alone without also tuning top-k gives an incomplete picture of retrieval quality.
A concrete method for tuning on real data
- Build a small evaluation set of 30 to 50 real questions with a known correct source chunk for each, drawn from actual user queries if available.
- Index the corpus at two or three candidate chunk sizes, keeping everything else constant.
- For each configuration, measure whether the correct source chunk appears in the top-k retrieved results, this is retrieval recall, not final answer quality yet.
- Compare recall across configurations and pick the smallest chunk size that does not measurably hurt recall, since smaller chunks generally cost less to embed and store.
- Only after settling on chunk size, tune overlap by testing whether increasing it recovers any of the boundary-cut failures observed in step three.
- Re-run this evaluation whenever the embedding model or a major new document type is introduced.
This method replaces guesswork with a measurable recall number tied to the specific corpus, which a fixed default number can never provide.
Common tuning mistakes that waste the exercise
Tuning chunk size against a synthetic or generic evaluation set, rather than real questions against the actual corpus, produces numbers that do not transfer to production. Similarly, tuning chunk size while ignoring the embedding model's effective context limit produces silently truncated chunks that never surface as an obvious error but quietly reduce recall. Treating overlap as a fixed percentage of chunk size without checking where actual answer-relevant sentences fall relative to chunk boundaries is another way to under-tune this parameter.
The evaluation set used to tune chunk size matters as much as the tuning method itself; a generic test set produces a generic, unreliable answer.
Frequently asked questions
Does a larger embedding model context window mean chunks should be larger?
Not necessarily. A larger context window means the model can represent longer text without truncation, but a very long chunk still produces a blurrier average-meaning vector, which can hurt retrieval precision even when the model technically supports the length.
Should overlap be a fixed number of tokens or a percentage of chunk size?
Either works; percentage-based overlap is more common because it scales naturally when chunk size changes during tuning, avoiding the need to re-derive an absolute token count each time.
Is there a chunk size that works well across nearly all document types?
A moderate size that balances specificity and context tends to generalize reasonably as a starting point across many document types, but "works reasonably" is different from "optimal," and only measurement against the real corpus confirms the latter.
How often should chunk size be re-evaluated after initial tuning?
Whenever the embedding model changes, a substantially different document type is added to the corpus, or retrieval quality metrics show a decline that other causes do not explain.
How Nanobase AI helps
Nanobase AI tunes chunk size and overlap against a client's actual documents and real query patterns rather than applying tutorial defaults, measuring retrieval recall directly before finalizing a configuration. This evaluation-driven approach is part of every RAG deployment we build. See the related chunking strategy by document type guide and our solutions for the broader pipeline.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.