Low vLLM throughput almost always traces back to one of a handful of common causes: gpu-memory-utilization set too conservatively low, which starves the KV cache of memory and caps how many sequences can batch together concurrently; max-num-seqs set too low for your traffic; prefix caching left disabled on a workload with repeated prompts; or tensor parallelism across GPUs connected by PCIe instead of NVLink, which turns required inter-GPU communication into a bottleneck. Another frequent mistake is benchmarking with a single sequential client, which never exercises continuous batching at all and produces throughput numbers far below what the engine achieves under real concurrent load. Start tuning by raising gpu-memory-utilization toward 0.9 to 0.95 if the GPU is dedicated to vLLM, enabling prefix caching if your prompts share any structure, and load testing at multiple concurrency levels with a tool like genai-perf to find where throughput actually saturates. Quantization to FP8 or AWQ can also raise throughput directly by reducing memory bandwidth pressure, which is usually the real bottleneck in decode-heavy workloads rather than raw compute. Check GPU utilization and NVLink bandwidth with nvidia-smi during a load test to confirm where the bottleneck actually sits before changing configuration blindly. Nanobase AI diagnoses these bottlenecks systematically as part of its performance tuning engagements.
Diagnose before you tune
The most expensive mistake in vLLM tuning is changing configuration before confirming where the actual bottleneck sits. A compute-bound prefill problem and a memory-bandwidth-bound decode problem call for different fixes entirely, and guessing wastes engineering time on changes that address the wrong layer. Before touching any flag, run nvidia-smi during a realistic load test to check GPU utilization and, on multi-GPU setups, NVLink bandwidth, since a GPU sitting at low utilization under load points somewhere very different than one pegged at 100 percent.
Confirm the actual bottleneck with GPU-level monitoring before changing any configuration, since the fix for a compute-bound problem does not help a memory-bandwidth-bound one.
Symptom-to-cause reference table
| Symptom | Likely cause | Fix to try |
|---|---|---|
| Throughput low even under many concurrent requests | gpu-memory-utilization set too conservatively | Raise toward 0.9–0.95 on a dedicated GPU |
| Low concurrency ceiling despite memory headroom | max-num-seqs set too low | Raise and re-test at target concurrency |
| Repeated prompts show no speed benefit | Prefix caching disabled | Enable prefix caching for shared prompt structure |
| Multi-GPU setup slower than expected | Tensor parallelism over PCIe instead of NVLink | Check interconnect; NVLink is required for TP to scale well |
| Benchmark numbers far below expectations | Testing with one sequential client | Load test with many concurrent clients (e.g. genai-perf) |
| Decode-heavy workload bottlenecked despite compute headroom | Memory bandwidth pressure, not compute | Apply FP8 quantization to weights and KV cache |
This table exists because most "vLLM is slow" reports resolve to one of these six causes, and matching symptom to cause correctly skips a lot of trial and error.
The benchmarking mistake that invalidates everything else
A single sequential client sending one request at a time and waiting for the response never exercises continuous batching at all, since there is nothing for the scheduler to batch. This produces a throughput number that reflects single-request latency, not the engine's actual concurrent-request capacity, and tuning against that number optimizes for the wrong thing entirely. Any throughput claim, from a vendor, a blog post, or your own prior testing, is only meaningful if it states the concurrency level it was measured at.
Load test at multiple concurrency levels with a tool built for this, such as genai-perf, to find where throughput actually saturates for your hardware and model, rather than relying on a single data point at unknown concurrency.
A throughput number without a stated concurrency level is not a usable benchmark, and single-client testing is the most common reason vLLM appears slower than it is.
Prefix caching: the free win most workloads leave on the table
Prefix caching reuses the computed KV cache for a shared prompt prefix (a system prompt, a repeated instruction template, a few-shot example set) across multiple requests instead of recomputing attention over that shared text every time. For workloads where a meaningful fraction of the prompt repeats across requests, chat applications with a fixed system prompt, RAG pipelines with a repeated instruction template, this cuts prefill compute substantially with no accuracy tradeoff at all, since the cached computation is exact, not approximate.
It has to be enabled explicitly and benefits scale with how much of the prompt is actually shared; a workload where every request's content is entirely unique gets little from it, while one built around a stable template gets a real and consistent gain.
Prefix caching is close to a free throughput gain for any workload with repeated prompt structure, and checking whether it is enabled should be an early step in any tuning pass.
Interconnect matters more than most tuning guides mention
Tensor parallelism across GPUs requires fast, frequent communication between devices for every layer's computation, which is exactly what NVLink is built for and exactly what PCIe struggles with at scale. A tensor-parallel deployment across GPUs connected only by PCIe can end up bottlenecked on that communication rather than on compute or memory bandwidth at all, producing throughput well below what the same GPUs would achieve with NVLink, or below what a single GPU without tensor parallelism might achieve on a smaller model.
Checking actual interconnect topology (nvidia-smi topo -m) before assuming tensor parallelism will scale linearly avoids a common and confusing tuning dead end.
Verify NVLink connectivity before relying on tensor parallelism to scale throughput, since PCIe-connected GPUs can turn multi-GPU scaling into a net loss.
Frequently asked questions
What is the fastest first check when vLLM throughput seems low?
Confirm the benchmark used realistic concurrent load, not a single sequential client, then check nvidia-smi during that load test to see whether the GPU is actually saturated; these two checks alone resolve a large share of reported throughput problems.
Does quantization always improve throughput?
For decode-heavy workloads, usually yes, since FP8 or AWQ reduces the memory bandwidth pressure that dominates token-by-token generation; for compute-bound prefill-heavy workloads the gain is smaller since compute, not bandwidth, is the constraint there.
How do I know if prefix caching is actually helping?
Compare time-to-first-token for a request with a fully unique prompt versus one sharing a cached prefix; a meaningful reduction for the cached case confirms it is working, and vLLM's metrics also expose cache hit rate directly.
Is there a throughput ceiling I should expect on a single H100?
This depends heavily on model size, quantization, and context length, so avoid anchoring to any generic published number; establish your own ceiling through load testing your specific model and traffic shape on your specific hardware.
How Nanobase AI helps
Nanobase AI diagnoses these bottlenecks systematically as part of its performance tuning engagements, confirming the actual constraint through GPU-level monitoring before applying any configuration change. This pairs with our deeper look at max-num-seqs and gpu-memory-utilization and our guidance on measuring LLM latency and throughput correctly.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.