A well-architected RAG system can handle enterprise document collections ranging from a few thousand files up to hundreds of millions of chunks, since the limiting factor is the vector database's indexing and query architecture rather than any inherent ceiling in the RAG approach itself. Small deployments in the tens of thousands of documents run comfortably on pgvector or a single-node vector database instance with sub-second query latency; mid-size corpora in the millions of chunks typically need a dedicated vector database like Qdrant or Milvus with HNSW indexing and adequate memory to hold the index; and very large deployments in the tens or hundreds of millions of vectors require horizontally sharded, distributed vector databases, quantization to control memory footprint, and careful attention to index build and update time. Beyond raw scale, retrieval quality tends to degrade as the corpus grows unless metadata filtering and hybrid search are used to narrow the candidate pool before similarity search, since a larger haystack makes it statistically more likely that near-duplicate or superficially similar irrelevant chunks appear in results. Document count alone is a poor predictor of difficulty compared to how heterogeneous, sensitive, and fast-changing the corpus is. Nanobase AI has architected RAG systems from single-department knowledge bases up to enterprise-wide document estates.
Scale is a staged problem, not a single ceiling
Asking how many documents a RAG system "can handle" implies a single ceiling, but the more accurate framing is that different corpus sizes call for different architectural choices, and a system built correctly at one tier migrates cleanly to the next rather than hitting a hard wall. The limiting factor throughout is the vector database's indexing and query architecture, not something inherent to the RAG approach itself, which is why the right question is which tier a given corpus falls into and what that tier's architecture should look like.
RAG scaling is a sequence of architectural tiers, each with a well-understood set of choices, rather than a single number where the approach stops working.
Scaling tiers at a glance
| Tier | Rough chunk count | Typical architecture |
|---|---|---|
| Small | A few thousand to hundreds of thousands | pgvector on a single Postgres instance, or a single-node vector database |
| Medium | Hundreds of thousands to low millions | Tuned pgvector with HNSW, or a single-node dedicated vector database like Qdrant |
| Large | Low millions to tens of millions | Dedicated vector database with replication and sharding, such as Qdrant or Milvus clustered |
| Very large | Tens of millions to hundreds of millions or more | Distributed vector database architecture with separate compute and storage layers, such as Milvus at cluster scale |
Each tier has a well-established set of tools; the mistake to avoid is over-architecting for a tier the corpus has not reached yet, or under-architecting once it has.
What actually breaks first as scale increases
Rather than a single point of failure, scale tends to strain the pipeline in a predictable order: index build and rebuild time grows first, becoming a scheduling constraint before it becomes a hard failure; query latency under concurrent load degrades next, especially for filtered queries; and finally, at the largest scales, a single node's memory can no longer hold the full index, forcing a move to a distributed or sharded architecture. Watching these three signals, rather than a raw chunk count, is the more reliable way to know when to move to the next tier.
Index build time, then concurrent query latency, then memory ceiling, is the typical order in which scale-related problems appear.
Retrieval quality does not automatically degrade with scale, but it can if untended
A common misconception is that retrieval accuracy inherently declines as the corpus grows, but a well-tuned approximate nearest neighbor index maintains consistent recall characteristics as it scales, when index parameters are re-tuned appropriately for the new size. What does decline without attention is precision, because a larger corpus increases the chance of near-duplicate or superficially similar chunks competing for the same top-k slots, which is a reason hybrid search and reranking become more valuable, not less, as a corpus grows.
Scale does not inherently hurt retrieval accuracy, but it does raise the value of hybrid search and reranking as more near-duplicate candidates compete for the same ranking slots.
Frequently asked questions
At what chunk count should we stop using pgvector?
There is no fixed number; the more reliable signals are HNSW index build time exceeding the available maintenance window, or concurrent query latency degrading under real traffic despite tuning, both of which tend to appear somewhere in the low millions of chunks for typical enterprise hardware.
Does adding more documents always require more GPU capacity?
Not directly for the vector search itself, which is often CPU-bound at moderate scale. It does increase the one-time and ongoing embedding compute needed to index new content, and a larger corpus with more concurrent users increases the reranking and generation workload.
Can a RAG system scale by simply adding more vector database nodes?
For vector databases designed for horizontal scaling, such as Milvus's distributed architecture, yes. For a single-node deployment like a standalone pgvector instance, scaling requires either vertical scaling of that instance or migrating to a horizontally scalable engine.
Is there a point where RAG stops being the right approach regardless of architecture?
Not due to scale itself; enterprise RAG systems handle very large corpora successfully with the right tier of architecture. The approach becomes questionable only when the actual problem is not a search problem, such as when every answer requires precise computation better served by a database query.
How Nanobase AI helps
Nanobase AI, an accepted member of the NVIDIA Inception Program, architects RAG systems for the scale tier a client is actually in today while designing the retrieval layer so migration to the next tier is a contained project, not a rewrite, as the corpus grows. See the related guide on RAG architecture for thousands of concurrent users and our solutions for the full infrastructure stack.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.