max-num-seqs sets the maximum number of sequences vLLM's scheduler will batch together concurrently at any point in the continuous batching loop, acting as a hard cap on concurrency regardless of how much GPU memory might technically be available for more, which protects against scheduling overhead and latency degradation from batching too many sequences at once. gpu-memory-utilization sets the fraction of total GPU memory vLLM is allowed to reserve when it starts up, covering model weights, activation memory, and most importantly the KV cache pool that continuous batching draws from, with the remainder left free for the CUDA driver, other processes, or headroom against out-of-memory errors. The two settings interact directly: raising gpu-memory-utilization gives vLLM a larger KV cache pool, which allows more sequences and longer contexts to fit simultaneously, but max-num-seqs still caps how many of those it will actually batch together even when memory allows more. A common tuning pattern is setting gpu-memory-utilization as high as safely possible for a dedicated GPU, typically 0.9 to 0.95, then adjusting max-num-seqs based on observed latency at your target concurrency rather than a generic default. Getting either one wrong is a common cause of lower than expected throughput. Nanobase AI, a Silicon Valley infrastructure company, tunes these two parameters as a pair against each customer's traffic shape.
Two settings, two different jobs
It helps to separate what each flag actually controls before tuning either one. gpu-memory-utilization sets the fraction of total GPU memory vLLM reserves at startup, covering weights, activation memory, and the KV cache pool, with the rest left for the CUDA driver and safety margin. max-num-seqs is a scheduler cap, the maximum number of sequences the continuous batching loop will process together at once, enforced regardless of how much memory is technically still free.
The interaction is straightforward once stated plainly: gpu-memory-utilization determines how big the KV cache pool is, and max-num-seqs determines how many sequences are allowed to draw from that pool concurrently, even if the pool has room for more. Raising one without the other only helps up to whichever ceiling the other setting imposes first.
One flag sizes the KV cache pool; the other caps how many sequences share it, and tuning only one while ignoring the other leaves throughput on the table.
A worked example on an 80 GB GPU
Consider a 70B model at FP8 (roughly 70 GB of weight memory) on an H100 with 80 GB total. At gpu-memory-utilization 0.9, vLLM reserves about 72 GB, leaving only about 2 GB for KV cache after weights, which is not enough for meaningful concurrency at any reasonable context length. This is a case where the model itself, not the settings, is the constraint, and the realistic fixes are a smaller model, a smaller quantization (INT4), or tensor parallelism across a second GPU to split the weight footprint and free real KV cache room.
Now consider the same GPU running a 13B model at FP8 (roughly 13 GB weights). At gpu-memory-utilization 0.9, about 72 GB is reserved, leaving roughly 59 GB for KV cache, activation memory, and everything else, comfortably supporting a high max-num-seqs value at moderate context lengths. Here, max-num-seqs is genuinely the tuning lever, since memory is not the binding constraint.
Whether max-num-seqs or GPU memory is your actual limiting factor depends entirely on how much room is left after model weights, which is why the same setting recommendation does not transfer across model sizes.
Related flags that shape the same tradeoff
| Flag | What it does | Interacts with |
|---|---|---|
max-num-seqs | Caps concurrent sequences in one batching step | KV cache pool size from gpu-memory-utilization |
max-num-batched-tokens | Caps total tokens processed in one scheduling step, across all sequences | Balances prefill and decode work within a batch |
enable-chunked-prefill | Splits large prefill work into chunks interleaved with decode steps | Reduces the chance a long prompt stalls ongoing decode work |
swap-space | CPU memory used to swap out sequences under memory pressure | Acts as a safety valve rather than a performance lever |
max-num-batched-tokens deserves particular attention alongside max-num-seqs, since a small number of very long sequences can consume as much scheduling budget as many short ones; capping total tokens per step, not just sequence count, prevents a handful of long requests from crowding out the batch.
max-num-seqs caps sequence count, but max-num-batched-tokens caps the actual token-level work per step, and a workload with variable sequence lengths needs both tuned together, not just one.
A practical tuning sequence
- Set
gpu-memory-utilizationto 0.9–0.95 for a dedicated GPU, confirming no other process shares it. - Check how much memory remains for KV cache after weights load, using vLLM's own startup logs, which report this breakdown.
- Estimate KV cache per sequence at your target context length (this scales with model size and context length, not batch size).
- Set
max-num-seqsto a value the remaining KV cache pool can actually support at that context length, rather than an arbitrary round number. - Load test at that setting and adjust based on observed latency, since the theoretical maximum concurrency and the concurrency that keeps latency acceptable are not always the same number.
Size max-num-seqs from the actual remaining KV cache budget after weights, not from a generic default, then validate the result under real load rather than trusting the arithmetic alone.
Frequently asked questions
What happens if max-num-seqs is set higher than memory can support?
vLLM will queue additional requests rather than crash, since the scheduler respects actual memory availability regardless of the configured cap; but this usually means the cap is not actually constraining anything and could safely be lowered for more predictable latency, or the memory pool needs to grow instead.
Does raising gpu-memory-utilization risk out-of-memory errors?
It reduces the safety margin reserved for the CUDA driver and other processes, so pushing it very close to 1.0 does raise that risk, particularly if another process shares the GPU; 0.9 to 0.95 is a common safe range for a fully dedicated GPU.
Is there a way to see how vLLM calculated its KV cache pool size?
Yes, vLLM logs this breakdown at startup, showing how many GPU memory blocks are available for KV cache after loading model weights, which is the most reliable input for sizing max-num-seqs correctly.
Should max-num-seqs differ between a chat workload and a batch summarization workload?
Yes; short interactive chat requests support a higher max-num-seqs at the same memory budget than long-document summarization requests, since KV cache per sequence scales with context length, so the right value is workload-specific rather than a universal default.
How Nanobase AI helps
Nanobase AI, a Silicon Valley infrastructure company, tunes these two parameters as a pair against each customer's traffic shape, using the actual KV cache budget from startup logs rather than generic defaults. This connects to our broader vLLM throughput tuning guide and GPU memory reduction guidance in this series.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.