There is no single number of concurrent users an H100 can serve, because it depends heavily on model size, average context length, output length, target latency and the quantization and batching strategy in use, but the ranges are informative. A small 7B to 8B model in FP8 with continuous batching can typically support on the order of dozens to over a hundred concurrent light chat sessions on one H100, while a 70B model, even in FP8, generally supports a much smaller number, often in the range of ten to thirty concurrent sessions, before KV cache and compute become the bottleneck. Long context windows and long generated responses both reduce this number substantially, since each active request holds its own KV cache for the duration of generation. Continuous batching in vLLM or TensorRT-LLM materially improves effective concurrency compared to naive request-by-request serving, and NVIDIA NIM microservices ship these optimizations by default. The only reliable way to know the real number for a given workload is to benchmark it under realistic traffic patterns rather than rely on a rule of thumb. Nanobase AI, headquartered in Silicon Valley, runs exactly this kind of load test before finalizing a GPU sizing recommendation.
Concurrency ranges by model size and quantization
There is no single correct number, but published benchmarks and production deployments cluster into recognizable ranges once model size, quantization and average context length are fixed. Treat these as starting points for your own load test, not guarantees.
| Model size | Precision | Avg. context | Approx. concurrent light chat sessions on 1x H100 |
|---|---|---|---|
| 7B-8B | FP8 | ~2K tokens | Dozens to over 100 |
| 7B-8B | FP8 | ~8K tokens | Roughly half of the above |
| 13B-14B | FP8 | ~2K tokens | Dozens |
| 32B-35B | FP8 | ~2K tokens | Roughly 15-30 |
| 70B | FP8 | ~2K tokens | Roughly 10-30 |
| 70B | FP8 | ~8K tokens | Roughly 5-15 |
Key takeaway: concurrency drops sharply as model size and context length both grow, since both increase memory and compute cost per request simultaneously.
The two resources that set the ceiling
Two distinct constraints determine how many concurrent users an H100 can serve, and they hit at different points. The first is memory: each active request holds its own KV cache, so once cumulative KV cache plus model weights approach 80 GB, no more requests can be admitted regardless of how much spare compute exists. The second is compute: even with memory available, the GPU's tensor cores can only process so many tokens per second across all active requests combined, so past a certain batch size, adding more concurrent sessions increases per-request latency rather than adding real capacity.
Continuous batching, used by vLLM and TensorRT-LLM, manages the transition between these two regimes automatically, admitting new requests as memory allows and adjusting the active batch as sessions complete, which is what makes the aggregate throughput numbers in production deployments meaningfully higher than naive sequential serving would achieve.
Key takeaway: memory sets the ceiling on how many requests can be active at once, while compute sets the ceiling on how fast they can all be served together.
How to test this on your own workload
- Deploy the target model with your intended quantization and serving engine on the actual GPU under consideration.
- Generate synthetic traffic matching your expected prompt length, output length and arrival pattern, rather than a uniform artificial load.
- Ramp concurrent sessions upward in steps, recording time-to-first-token and steady-state tokens per second at each step.
- Identify the point where latency degrades past your acceptable threshold; that is your practical concurrency ceiling, not the point where the GPU runs out of memory.
- Repeat with a quantized KV cache and with a higher
gpu-memory-utilizationsetting to see whether either buys meaningful additional headroom.
Key takeaway: the only number that matters is the one measured on your actual model, prompts and latency target, since published ranges vary by a wide margin.
How H200 shifts the same ranges
Because the H200 shares H100's compute but offers 141 GB of memory against 80 GB, the memory-bound ceiling in the table above moves outward while the compute-bound ceiling stays roughly the same. In practice this means a 70B model's concurrency range on an H200 extends further into the upper end of what compute alone would allow, since KV cache is no longer the first thing to run out; a deployment that hits 15 concurrent 8K-context sessions on an H100 due to memory can often push meaningfully higher on an H200 before compute becomes the binding constraint instead.
Key takeaway: H200's extra memory raises the concurrency ceiling for memory-bound workloads like 70B chat, but does not change the compute ceiling for the same model. For a worked example at a specific target, see sizing GPUs for 100 concurrent chatbot users.
Frequently asked questions
Why do published H100 concurrency numbers vary so much between sources?
Because they depend on assumptions rarely stated clearly: average prompt length, output length, acceptable latency, quantization, and whether the figure is peak burst capacity or sustained throughput. Two benchmarks using the same model can report very different numbers if these assumptions differ.
Does NVIDIA NIM change these numbers?
NIM microservices ship with NVIDIA's own serving optimizations and continuous batching by default, which typically puts throughput near the upper end of what a well-tuned vLLM or TensorRT-LLM deployment achieves, without requiring manual tuning.
Is time-to-first-token or tokens-per-second the better concurrency metric?
Both matter for different reasons. Time-to-first-token reflects how responsive the system feels to a user sending a new message, while steady-state tokens per second reflects how quickly a response streams once started; a system can look fine on one and poor on the other under high concurrency.
Should I size for average or peak concurrency?
Peak. Chat and assistant traffic is rarely evenly distributed across the day, so sizing to an average concurrency figure routinely results in degraded performance during the busiest periods, which is when the deployment's reputation is made or broken.
How Nanobase AI helps
Nanobase AI, headquartered in Silicon Valley, runs exactly this kind of load test, ramping concurrency against a target model and serving engine on candidate GPUs, before finalizing a sizing recommendation for a customer. We measure both time-to-first-token and steady-state throughput against your acceptable latency, not just a peak concurrency number, and detail the engine trade-offs further in our vLLM vs TensorRT-LLM vs Ollama vs SGLang comparison.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.