How many GPUs do I need for 100 concurrent chatbot users?

For 100 concurrent chatbot users, the GPU count depends primarily on model size and average conversation length, but as a starting point, a 7B to 8B model in FP8 with continuous batching often handles 100 concurrent light chat sessions on a single H100 or H200, while a 70B model typically needs two to four H100 or H200 GPUs with tensor parallelism to sustain the same concurrency at acceptable latency, since each session's KV cache and the larger compute cost per token both reduce how many requests one GPU can serve well. Longer average messages, larger context windows, or a requirement for very low time-to-first-token all push the estimate toward more GPUs, while quantizing to INT4 or FP8 and enabling KV cache quantization can pull it back down. Peak concurrency also matters more than average concurrency, since chatbot usage is rarely evenly distributed across the day, so sizing for the busiest hour rather than the daily average avoids degraded performance under real load. A realistic plan builds in headroom above the measured concurrency target rather than sizing to the exact number. Nanobase AI benchmarks the target chatbot workload on candidate GPU configurations before recommending a final count.

Sizing table for 100 concurrent sessions

The GPU count for 100 concurrent chatbot users depends primarily on model size, quantization and average context length. The table below assumes continuous batching and short-to-moderate chat context (roughly 2K to 4K tokens per session).

Key takeaway: the same 100-user target can need anywhere from one GPU to eight depending on model size and average context, so the model choice matters more than the user count.

Why the H200 changes the calculus at 70B

For a 70B model, the jump from FP8 weights (70 GB) leaves comparatively little headroom on an 80 GB H100 for the KV cache that 100 concurrent sessions generate, which is why the table above shows a wide GPU count depending on context length. An H200's 141 GB roughly doubles that headroom per GPU, so fewer H200s are typically needed to hit the same 100-user target than H100s would require, even though each H200 costs more individually. Whether the H200's premium is worth it versus simply adding more H100s depends on rack space, power budget and how much future growth headroom the deployment needs, since fewer, larger GPUs also means less tensor-parallel communication overhead per request.

Key takeaway: an H200-based configuration for 100 concurrent 70B users often needs fewer total GPUs than an H100-based one, trading GPU count for a higher per-GPU cost.

Building in headroom instead of sizing to the exact number

1. Take the measured or estimated GPU count for 100 concurrent users at your target model and context length.
2. Add 20 to 30 percent headroom to absorb usage spikes, since chatbot traffic is rarely perfectly smooth even at a stated concurrency target.
3. Confirm the serving engine's max-model-len and memory utilization settings are configured for the actual context distribution, not just the average.
4. Load-test at 100 concurrent sessions plus the headroom margin before committing to the final GPU count, since real traffic patterns rarely match synthetic assumptions exactly.
5. Re-evaluate quarterly as usage grows, since chatbot adoption inside an organization typically increases after a successful initial rollout.

Key takeaway: sizing to the exact estimated GPU count with no margin is the most common reason a chatbot deployment that tested fine degrades once real usage begins.

INT4 as a way to hit 100 users on fewer GPUs

The INT4 row in the table above shows a meaningful reduction versus FP8 at 70B, since shrinking weights from 70 GB to about 38 GB frees enough headroom to absorb 100 concurrent sessions' KV cache on fewer cards. This trade only makes sense once output quality has been validated against your own chatbot's representative queries, since a customer-facing chatbot at high volume is exactly the kind of deployment where a quality regression is costly to discover after launch rather than before. For deployments where FP8 quality is already borderline acceptable, INT4 is worth testing specifically for its GPU-count reduction, not just its raw memory saving.

Key takeaway: INT4 can cut the GPU count for a 100-user 70B chatbot roughly in half versus FP8, provided output quality has been validated first.

Frequently asked questions

Does "100 concurrent users" mean 100 people typing at the exact same moment?

Effectively yes for sizing purposes; it means 100 active generation requests in flight simultaneously, which is the number that determines KV cache and compute load, regardless of how many total registered users exist behind that concurrency figure.

Is a smaller model with more GPUs ever better than a larger model with fewer?

Sometimes, particularly if response quality from the smaller model is acceptable for the use case, since a smaller model at high concurrency can be cheaper to run overall than a larger model even accounting for the extra GPUs needed to hit the same concurrency target. This is worth testing rather than assuming.

How does output length affect the 100-user GPU count?

Longer average responses hold each request's KV cache active for longer and add more generated tokens to compute, both of which reduce how many concurrent sessions a fixed GPU count can sustain, so a chatbot with long-form answers needs more GPUs than one with short replies at the same user count.

Can NVIDIA NIM reduce the GPU count needed for 100 users?

NIM microservices ship with tuned continuous batching and serving optimizations out of the box, which can improve effective throughput compared to an unoptimized deployment, though the underlying memory and compute limits of the GPU itself remain the same regardless of serving software.

How Nanobase AI helps

Nanobase AI benchmarks the target chatbot workload, including realistic prompt and response length distributions, on candidate GPU configurations before recommending a final count for a 100-user or any other concurrency target. We also build in the headroom margin appropriate to your growth expectations rather than sizing to the exact measured number. See our live demo or read our guide to on-premise LLM deployment for the full process.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.