A RAG system for 500 users generally does not need GPU capacity sized to all 500 users being active simultaneously, since typical usage patterns put peak concurrency well below total headcount, often in the range of 25 to 75 simultaneous sessions for this size of user base depending on the use case. Given that concurrency range, a generation model in the 7B to 32B class running in FP8 or INT4 on one to two H100, H200 or RTX PRO 6000 GPUs typically covers both the generation workload and, if capacity allows, the smaller embedding and reranking models on the same or an additional modest GPU. The generation model is usually the GPU bottleneck in a RAG pipeline rather than retrieval, since vector search runs on CPU or a separate database and embedding or reranking models are comparatively lightweight, so most of the sizing budget should go toward the LLM serving layer. Larger context windows from retrieved document chunks add to KV cache usage more than people expect, since RAG prompts are often longer than typical chat messages once retrieved passages are included. Nanobase AI sizes the generation, embedding and reranking components of a RAG deployment together against real query volume.
Sizing the whole pipeline, not just the model everyone thinks about
A RAG deployment has at least three GPU-relevant components: the generation model, the embedding model used for retrieval, and often a reranker that re-scores retrieved candidates before they reach the generation model. Teams sizing a RAG system for the first time frequently size only the generation model and treat the rest as free, which works out approximately right given how small embedding and reranking models are, but skipping the explicit step means context-length assumptions from retrieved passages get missed too. A RAG prompt is rarely just the user's question; it typically includes several retrieved passages, which makes RAG requests longer and more KV cache-hungry than an equivalent plain chat message.
A worked capacity budget for 500 users
| Component | Sizing basis | Typical allocation |
|---|---|---|
| Peak concurrent sessions | 25–75 of 500 total users, workload-dependent | Drives generation model sizing |
| Generation model | 7B–32B class, FP8 or INT4 | 1–2 H100, H200 or RTX PRO 6000 |
| Embedding model | Small, throughput-bound not memory-bound | Co-located or shared spare capacity |
| Reranker (if used) | Small, batch-oriented | Co-located or shared spare capacity |
| Retrieved-context overhead | Adds to KV cache per request | Budget 500–2,000+ extra tokens per turn |
The concurrency range in the first row, not the 500-user total, is the number that should drive the generation model's GPU count; sizing directly to total headcount is the most common source of over-provisioning in RAG deployments specifically, since retrieval-augmented tools are typically used in bursts rather than continuously by every user at once.
Step-by-step sizing methodology
- Estimate realistic peak concurrency, using existing usage data for a similar internal tool if available, or a conservative percentage of headcount (commonly 5 to 15 percent for internal knowledge-worker tools) if not.
- Choose the generation model and precision, since this is the primary GPU cost driver in the entire pipeline; see how to calculate GPU memory for an LLM for the base formula.
- Add retrieved-context overhead to the KV cache estimate, treating a typical RAG turn as meaningfully longer than a plain chat turn to avoid underestimating memory per session.
- Size embedding and reranking capacity last, generally as co-located or shared GPU capacity given their modest footprint, detailed in VRAM for embedding models and rerankers.
- Load-test the full pipeline together, not the generation model in isolation, since end-to-end latency includes retrieval and reranking time that a generation-only benchmark will miss.
Where the real bottleneck usually sits
Vector search itself typically runs on CPU or a dedicated vector database and rarely becomes the GPU bottleneck in a RAG pipeline; the generation model consistently is. This means most of the sizing budget and most of the tuning effort, quantization level, batch configuration, KV cache management, should go toward the generation layer rather than splitting attention evenly across all three components. For 500 users, one to two well-chosen GPUs running the generation model in FP8 or INT4 covers the large majority of realistic RAG deployments, with embedding and reranking riding along on the same or a modest additional card.
Frequently asked questions
Does a bigger user base always mean more GPUs for RAG?
Not linearly. Peak concurrency, not headcount, drives GPU sizing, so a 500-user deployment with low simultaneous usage can need less hardware than a 100-user deployment where most users are active at once, such as a customer-facing support tool during business hours.
Should the generation model be a large model like a 70B for RAG, or is smaller enough?
Many RAG use cases perform well with a 7B to 32B generation model since retrieval supplies the specific facts, reducing reliance on the model's own parametric knowledge; whether a larger model is worth the extra hardware depends on how much reasoning versus pure retrieval-based answering the use case requires.
How much does retrieved context add to GPU memory needs?
Each retrieved passage typically adds a few hundred to over a thousand tokens depending on chunk size and how many chunks are retrieved per query, which should be added explicitly to the KV cache estimate rather than assumed away, since it can meaningfully exceed the length of the user's actual question.
Can the embedding and reranking models share a GPU with the generation model?
Yes, in most cases, given their combined footprint of typically well under 10 GB, this is the most cost-effective placement when the generation model's precision and quantization leave that much headroom on the same card.
How Nanobase AI helps
Nanobase AI sizes the generation, embedding and reranking components of a RAG deployment together against real query volume, rather than treating the generation model as the whole picture. Explore the broader trade-offs in RAG vs. fine-tuning or see the solutions Nanobase AI builds around retrieval-augmented deployments.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.