Why does my LLM slow down or fail when many users connect at once?

An LLM typically slows down or fails under many simultaneous users because the KV cache, which grows with both the number of active sessions and their context length, fills the GPU's available memory, forcing the serving engine to queue new requests, reduce batch size, or in poorly configured setups run out of memory entirely. As the KV cache pool approaches capacity, engines like vLLM will delay accepting new requests rather than crash, which shows up to users as rising latency or timeouts rather than an obvious error, making the root cause easy to miss without memory monitoring in place. Long conversations, large system prompts, or users pasting large documents into a chat window all increase per-session KV cache usage far more than short exchanges do, so a system tuned for brief queries can degrade sharply once usage patterns shift. The fix is usually some combination of adding GPU capacity, reducing maximum context length, quantizing the KV cache, or setting stricter per-request limits rather than trying to serve unlimited context to unlimited concurrent users on fixed hardware. Monitoring GPU memory utilization and queue depth in production surfaces this problem before it becomes visible to end users. Nanobase AI diagnoses and resolves exactly this class of concurrency bottleneck for customers already in production.

How a serving engine actually behaves at the memory ceiling

Modern serving engines like vLLM and TensorRT-LLM use continuous batching: instead of waiting for a fixed batch to finish, they admit new requests into a running batch token by token, and each admitted request reserves a slab of KV cache for its current and future tokens. This is what makes GPU utilization high under normal load, but it also means the engine's admission decision is a direct function of free KV cache pages, not raw GPU compute. When free pages run low, the scheduler does not crash or reject work outright; it delays admitting new sequences and may even preempt or swap out lower-priority ones, which is why the symptom users see is rising latency and stalled responses rather than a clean error message.

The slowdown is a scheduling response to memory pressure, not a compute bottleneck, which is why adding more compute-focused optimizations rarely fixes it and adding memory or shrinking memory demand usually does.

Sizing the KV cache budget with a worked example

KV cache size per token follows a fixed formula: 2 (one tensor for keys, one for values) × number of layers × number of KV heads × head dimension × bytes per element. Grouped-query attention, used in most current open models, keeps the KV head count much lower than the query head count specifically to shrink this number. For illustration, a 32B-class dense model with 64 layers, 8 KV heads and a 128 head dimension in FP16 works out to roughly 0.25 MB of KV cache per token per active sequence. That per-token cost is what multiplies across every concurrent session and every token of context each one holds.

On an 80 GB H100 already holding a 32B model's weights at roughly 64 GB in FP16, there is no room left for the 50-session row, let alone 200. This table is the reason a deployment that felt fine in testing with five users can fail publicly at fifty: KV cache demand scales linearly with both concurrency and context length, and it competes directly with the weights for the same fixed pool of HBM.

The warning signs that appear before users complain

GPU memory utilization climbing toward the engine's configured ceiling (commonly 90 to 95 percent of available memory) is the earliest signal, followed by growing request queue depth and a widening gap between time-to-first-token and the model's normal latency under light load. Watching only average latency hides this, since the early requests in a surge still complete quickly while later ones queue; percentile latency (p95, p99) and queue depth are the metrics that catch the problem before a support ticket does.

Five ways to raise the concurrency ceiling

1. Quantize the KV cache itself (FP8 KV cache is supported by vLLM and TensorRT-LLM) to roughly halve its footprint independent of the model's weight precision.
2. Cap maximum context length per request so a handful of long conversations cannot starve capacity meant for many short ones.
3. Add admission control or a request queue with a visible wait state, which turns an invisible failure into a predictable, user-facing delay.
4. Lower gpu_memory_utilization deliberately during testing, then raise it only once real traffic patterns are measured, rather than assuming the full advertised VRAM is usable.
5. Add a second GPU or move to a higher-memory card (H200 or B200) when the above changes still cannot cover peak measured concurrency.

Understanding these levers before an incident, rather than during one, is the difference between a planned upgrade and an emergency one; the how many GPUs for 405B and RAG sizing articles walk through the same KV cache budgeting for larger deployments.

Frequently asked questions

Does adding more GPUs always fix concurrency-related slowdowns?

Not automatically. Additional GPUs help only if the deployment is configured to use them for KV cache capacity, either through data parallelism (replicating the model to serve more independent sessions) or a genuinely larger combined memory pool. Simply adding a GPU without adjusting the serving topology can leave the bottleneck unchanged.

Is this the same problem as running out of GPU compute?

No. Compute-bound slowdowns show up as consistently high GPU utilization with steady throughput; memory-bound concurrency slowdowns show up as growing queue depth and stalled admissions while compute utilization may look moderate. Distinguishing the two determines whether the fix is more GPUs, better batching, or a KV cache change.

Can I predict my concurrency ceiling before going to production?

Yes, approximately, using the KV cache formula above combined with expected average context length, though the only reliable number comes from load-testing the actual model and serving engine configuration against realistic traffic before launch.

How Nanobase AI helps

Nanobase AI diagnoses concurrency bottlenecks in production deployments and designs the KV cache budget, admission control and GPU capacity upfront so they do not appear as an incident later. That includes benchmarking the actual model and traffic pattern, tuning serving engine memory parameters, and recommending the right point to add GPU capacity versus the right point to change configuration instead.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.