A RAG architecture serving thousands of concurrent users needs horizontal scaling at every stage of the pipeline, not just a larger language model, since each concurrent query triggers an embedding call, a vector database query, often a reranking call, and a language model generation call, and any one of those stages can become the bottleneck under load. The embedding and reranking models should run behind a dedicated inference server such as vLLM or NVIDIA Triton with dynamic batching, so many concurrent requests share GPU compute efficiently rather than each query blocking on its own inference call. The vector database needs to be sharded or replicated across nodes with enough memory to keep the index resident rather than reading from disk under load, since disk-bound vector search latency degrades sharply under concurrent query volume. Caching frequently asked questions and their retrieved chunks, or even full generated answers for identical queries, meaningfully reduces load for common questions in a typical enterprise knowledge base where query patterns repeat heavily. The language model serving layer needs autoscaling GPU capacity, typically on H100 or H200 infrastructure with a Kubernetes-based orchestration layer, to absorb traffic spikes without queuing delays that users notice. Nanobase AI architects and load-tests RAG systems against realistic concurrent user projections before production rollout.
Every stage needs its own capacity plan, not just the model
Scaling a RAG system to thousands of concurrent users fails when one team assumes a bigger, faster language model solves the whole problem, because every query also triggers an embedding call, a vector database lookup, and often a reranking call, and any one of these four stages can become the bottleneck independently of the others. A capacity plan needs a scaling lever and a monitoring metric for each stage individually, since a system tuned only for language model throughput can still collapse under load if the vector database or embedding service saturates first. Capacity planning for RAG at scale means identifying the bottleneck stage by stage, since the language model is only one of four stages that can saturate under concurrent load.
The capacity plan by pipeline stage
| Stage | Common bottleneck | Scaling lever | Monitoring metric |
|---|---|---|---|
| Embedding inference | GPU throughput at high concurrent query volume | Dedicated inference server (vLLM, Triton) with dynamic batching | Embedding latency p99, queue depth |
| Vector database query | Disk-bound search once the index exceeds available memory | Sharding or replication, ensure index stays memory-resident | Query latency p99, memory utilization |
| Reranking inference | Similar to embedding, GPU-bound at high concurrency | Dedicated inference server, batch reranking requests | Reranking latency p99, GPU utilization |
| Language model generation | GPU capacity for concurrent generation requests | Autoscaling GPU pool, typically H100 or H200 based | Time-to-first-token, tokens per second, queue wait time |
| Overall pipeline | Any single stage above, or network overhead between them | Caching frequent queries and answers | End-to-end latency p50 and p99 |
Key takeaway: monitor latency and utilization at each of the four stages independently, since aggregate end-to-end latency alone will not reveal which stage is actually saturating first.
A load-testing methodology that finds the real bottleneck
- Define a realistic concurrent user target and query pattern based on expected usage, not a round number, since a support-desk use case with short factual queries stresses the pipeline very differently than a research use case with long, complex questions.
- Run load tests against each pipeline stage in isolation first, embedding, vector search, reranking, and generation separately, to establish each stage's individual capacity ceiling before testing them together.
- Run the full pipeline under increasing concurrent load, tracking latency and error rate per stage, not just the overall response time, to identify which stage's latency degrades first as load increases.
- Test cache effectiveness specifically, since enterprise knowledge base query patterns often repeat heavily, and a well-tuned cache for frequently asked questions can meaningfully reduce the load reaching the more expensive downstream stages.
- Re-run the full load test after any capacity change, such as adding GPU replicas or sharding the vector database, to confirm the fix actually raised the ceiling rather than just shifting the bottleneck to a different stage.
Key takeaway: isolating each pipeline stage during load testing identifies the actual first bottleneck, which is rarely obvious from watching overall response time alone.
Why the vector database is an easy stage to underestimate
Teams sizing a RAG architecture for scale often focus disproportionately on GPU capacity for the language model, the most visible cost line item, and underestimate the vector database's own scaling needs. An index that fits comfortably in memory at small scale can silently become disk-bound as document volume grows, and disk-bound search latency degrades sharply under concurrent load in a way that is easy to miss during light testing but severe once thousands of users query simultaneously. Sizing memory for projected document growth over the deployment's lifetime, not launch-day volume, avoids this becoming a mid-life surprise; see the Kubernetes vector database comparison for more.
Key takeaway: size vector database memory for projected future document volume, not launch-day volume, since disk-bound search latency under concurrent load is a common and avoidable scaling failure.
Frequently asked questions
Does caching full generated answers risk serving stale or wrong content?
Yes, if not managed carefully; cached answers should have a defined expiration tied to the same re-indexing cadence used for the underlying documents, and a cache invalidation trigger when a cached query's source documents change, rather than an indefinite cache that can serve an outdated answer.
How much does GPU choice, like H100 versus H200, matter for this kind of scaling?
It matters most at the generation stage, where H200's larger 141 GB HBM3e memory and 4.8 TB/s bandwidth support higher concurrent batch sizes than an H100's 80 GB HBM3 and 3.35 TB/s, covered in detail in our H100 vs H200 vs B200 comparison, though embedding and reranking stages typically need less GPU memory per instance.
Should embedding and reranking share GPU capacity with the main language model?
They can, using techniques like NVIDIA's Multi-Instance GPU to partition a single card, but at genuinely high concurrency, dedicating separate GPU capacity to each stage usually gives more predictable latency than sharing, since contention between stages becomes harder to reason about under heavy load.
What is a reasonable target for end-to-end latency at scale?
There is no universal target, since it depends on the use case, but tracking p99 latency, not just average, matters more at scale, since a system with a good average but a long tail of slow responses will still generate visible user complaints during peak concurrent load.
How Nanobase AI helps
Nanobase AI architects and load-tests RAG systems stage by stage against realistic concurrent user projections before production rollout, sizing GPU capacity, vector database memory, and caching together rather than optimizing any one stage in isolation. See our solutions for the full architecture.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.