There is no single ideal top-k for every RAG system, but a common and reasonable starting point is retrieving 20 to 50 candidates through initial vector or hybrid search, reranking them, and sending only the top 3 to 8 chunks to the language model after reranking, since sending too few chunks risks missing the answer while sending too many dilutes the model's attention and increases the chance it blends or confuses unrelated content. The right number depends on chunk size and the language model's context handling: smaller chunks in the 200 to 400 token range often work well with a top-k of 5 to 10 after reranking, while larger chunks closer to 800 tokens usually need a smaller top-k, around 3 to 5, to avoid overwhelming the context with redundant information. Questions requiring synthesis across multiple sources, such as comparing figures across several documents, generally need a higher top-k than simple factual lookups that a single well-matched chunk can answer completely. This setting should be tuned against a labeled evaluation set measuring both answer accuracy and irrelevant-chunk rate, since more context is not automatically better and often measurably hurts both accuracy and cost past a certain point. Nanobase AI tunes top-k empirically per use case rather than applying a single fixed default across every RAG deployment.
Top-k is not an independent setting
Treating top-k as a single number to tune in isolation misses that it interacts directly with two other decisions: chunk size and whether a reranker sits between initial retrieval and final selection. The same top-k value behaves completely differently depending on whether chunks are 200 tokens or 800 tokens, and reranking changes what top-k after reranking should even mean compared to top-k straight out of vector search. Top-k only makes sense as a setting when tuned alongside chunk size and the presence of a reranker, not as an isolated number.
How chunk size and top-k should move together
| Chunk size | Recommended top-k after reranking | Reasoning |
|---|---|---|
| 200-400 tokens | 5-10 | Smaller chunks need more of them to cover a complete answer, but each adds less noise individually |
| 500-800 tokens | 3-5 | Larger chunks carry more context per chunk, so fewer are needed and more risk diluting attention if over-included |
| Mixed or section-based chunking | 3-8, tuned per document type | Variable chunk sizes need per-corpus tuning rather than one fixed rule |
Key takeaway: smaller chunks generally need a higher top-k to cover a complete answer, while larger chunks need a lower top-k to avoid overwhelming the model with redundant context.
Why the initial retrieval count matters separately from the final top-k
The count of candidates pulled from the initial vector or hybrid search, commonly 20 to 50, is a separate setting from the final number sent to the model after reranking, commonly 3 to 8, and conflating the two is a common mistake. Pulling too few initial candidates risks missing the correct chunk before reranking can find it, since a reranker only reorders what retrieval already surfaced; pulling far more than necessary adds latency and cost without improving accuracy once the correct chunk is reliably in the candidate set. Key takeaway: initial retrieval count determines whether the right chunk is available to find, while final top-k determines how much of it reaches the model, and both need independent tuning.
A tuning procedure using a labeled set
- Fix chunk size and the presence or absence of a reranker as the two other variables, then vary only the final top-k against a golden test set, measuring both answer accuracy and the rate of irrelevant chunks appearing in context.
- Identify the point where increasing top-k stops improving accuracy and starts degrading it, which is common once redundant or tangential chunks begin competing with the correct one for the model's attention.
- Repeat the same sweep at two or three different chunk sizes to see whether the optimal top-k shifts, confirming the relationship rather than assuming it from the general guidance above.
- Test with and without a reranker at the top-k values found, since a reranker often allows a meaningfully lower final top-k while maintaining or improving accuracy, since less irrelevant content survives to reach the model.
- Segment the evaluation by question type where possible, since synthesis questions requiring information from multiple sources generally benefit from a higher top-k than simple single-fact lookups, and averaging across both question types can hide this difference.
Key takeaway: sweep top-k against a labeled set while holding chunk size fixed, then repeat at a different chunk size, rather than tuning the two settings independently of each other.
Question type changes the right answer meaningfully
A simple factual lookup, such as a single policy threshold, is usually answered completely by one well-matched chunk, and a higher top-k adds risk without benefit. A synthesis question combining figures from several source documents genuinely needs more chunks in context, and an overly conservative top-k produces an incomplete answer for exactly this question type even though it performs well on simple lookups. A system serving both well sometimes benefits from a lightweight query classifier that adjusts top-k dynamically, similar in spirit to the query-type awareness that separates semantic search from RAG in the first place.
Key takeaway: a fixed top-k tuned only against simple factual questions will underperform on synthesis questions that genuinely need more retrieved context.
Frequently asked questions
Does a higher top-k always increase cost proportionally?
Yes, more chunks in context means more input tokens sent to the language model, which increases both cost and generation latency, so top-k tuning is a cost decision as well as an accuracy one, and the optimal value from an accuracy-only sweep should still be checked against its cost impact.
Can dynamic top-k based on query type actually be implemented reliably?
Yes, using a lightweight classifier or even simple heuristics, such as query length or the presence of comparison language like "compare" or "across," to route queries toward a higher or lower top-k, though this adds a component that itself needs testing and monitoring rather than being a free improvement.
Is there a maximum top-k beyond which more context always hurts?
There is no universal number, since it depends on chunk size, the language model's own context handling ability, and how distinguishable relevant from irrelevant chunks are within a specific corpus, which is exactly why tuning against a labeled set for your own data matters more than any generic default.
Should top-k be the same for every document type in a mixed corpus?
Not necessarily; a corpus mixing short policy snippets with long technical manuals often benefits from per-document-type or per-chunk-size top-k tuning rather than one global value, especially if chunk sizes vary meaningfully across document types.
How Nanobase AI helps
Nanobase AI tunes top-k, chunk size, and reranking together empirically per use case against a labeled evaluation set, rather than applying a single fixed default across every RAG deployment. This tuning work is part of the broader diagnostic process we run when investigating irrelevant RAG results. See our solutions for the full pipeline.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.