How do I get vLLM to use less GPU memory?

Reducing vLLM's GPU memory usage starts with lowering the gpu-memory-utilization flag, which controls what fraction of total GPU memory vLLM reserves upfront for weights, activations, and KV cache, since the default is often set aggressively high assuming the GPU is dedicated to that one process. Quantizing the model to AWQ, GPTQ, or FP8 shrinks the weight footprint substantially, roughly halving memory for FP8 versus FP16 on Hopper and Blackwell GPUs, freeing that memory for KV cache and larger batch sizes instead. Lowering max-model-len to the context length your application actually needs, rather than the model's maximum supported length, directly reduces per-sequence KV cache allocation, and enabling FP8 KV cache quantization roughly halves KV cache memory on top of that with minimal accuracy impact on modern hardware. If a single GPU still cannot hold the model comfortably, tensor parallelism across two or more GPUs splits both weights and KV cache, trading GPU count for headroom per device. Reducing max-num-seqs caps how many concurrent sequences vLLM will batch, which lowers peak memory usage at the cost of some throughput under high concurrency. Nanobase AI tunes these memory parameters together against real workload profiles rather than adjusting one setting in isolation.

Where vLLM's memory actually goes

Before changing any flag, it helps to know what is competing for GPU memory in the first place. vLLM's memory budget splits roughly into four categories, and knowing which one is under pressure changes which fix actually helps.

An out-of-memory error during model loading points at weights; one that appears only under concurrent load points at KV cache. Applying a weight-focused fix (like quantization) to a KV-cache-driven OOM helps only indirectly, by freeing memory the cache can then use, while a cache-focused fix (like FP8 KV cache) does nothing for a load-time OOM caused by weights alone.

Diagnose which memory category is actually under pressure before picking a fix, since the four categories respond to different levers.

The order of operations that fixes most cases

1. Lower gpu-memory-utilization if it is set too aggressively for a shared GPU, or raise it toward 0.9–0.95 if the GPU is dedicated and you actually want more headroom for KV cache, not less.
2. Quantize the model to AWQ, GPTQ, or FP8, roughly halving weight memory for FP8 versus FP16 on Hopper and Blackwell hardware, freeing that space for KV cache and larger batches.
3. Cap max-model-len to the context length your application genuinely needs rather than the model's maximum supported length, since KV cache allocation scales directly with this value per sequence.
4. Enable FP8 KV cache (kv-cache-dtype fp8), roughly halving KV cache memory on top of weight quantization, with minimal accuracy impact on modern hardware.
5. Reduce max-num-seqs if concurrency, not context length, is driving cache size, accepting some throughput cost in exchange for headroom.
6. Add tensor parallelism across two or more GPUs as a last resort, splitting both weights and KV cache across devices when a single GPU genuinely cannot hold the workload even after the steps above.

Work through this list in order; each step targets a different memory category, and skipping straight to tensor parallelism (step 6) often masks a fixable single-GPU configuration problem instead of solving it.

Quantifying the two biggest levers

Quantization and context length capping deserve special attention because they typically produce the largest single gains. Moving a 70B model from FP16 to FP8 weights alone frees roughly half the weight memory footprint (from about 140 GB to about 70 GB), and that freed space converts directly into either more concurrent sequences or longer supported context, whichever your workload needs more. Separately, if an application only ever needs 8K tokens of context but max-model-len is left at the model's 128K maximum, every allocated sequence slot reserves cache space for context it will never use, which is pure waste at scale.

Quantization and a realistic max-model-len setting together usually free more memory than any single other change on this list, and both are configuration changes rather than architecture changes.

When tensor parallelism is the right call, not a workaround

Tensor parallelism is not a failure mode to avoid, it is the correct answer once a model genuinely does not fit a single GPU even after quantization and context capping, most commonly for models in the 70B-plus range on GPUs below 80 GB, or any workload needing both large batch size and long context simultaneously. The distinction worth making is that reaching for it before working through steps 1 through 5 above often means paying for a second GPU to solve a problem that better configuration on one GPU would have solved for free.

Use tensor parallelism when the workload genuinely exceeds one GPU's capacity, not as the first response to an out-of-memory error that better configuration could resolve.

Frequently asked questions

What is a safe starting value for gpu-memory-utilization?

0.9 is a common starting point for a GPU dedicated entirely to vLLM, leaving roughly 10 percent headroom for the CUDA driver and safety margin; lower it if the GPU shares memory with other processes.

Does FP8 quantization hurt output quality noticeably?

For most workloads, no; FP8 on Hopper and Blackwell hardware typically preserves quality close to FP16 for both weights and KV cache. Validate on your own evaluation set for tasks especially sensitive to precision, such as complex numerical reasoning.

Why does vLLM still run out of memory even after lowering max-num-seqs?

Check whether max-model-len is set far above what requests actually use, since a high ceiling reserves cache space per sequence slot regardless of concurrency; also confirm no other process is competing for the same GPU's memory.

Is it worth quantizing to INT4 for a memory-constrained deployment?

Yes for models that would otherwise not fit at all, since INT4 shrinks weights to roughly a quarter of FP16 size, though expect a larger quality tradeoff than FP8 and validate carefully on your own tasks before committing to it in production.

How Nanobase AI helps

Nanobase AI, a Silicon Valley enterprise AI engineering company, tunes these memory parameters together against real workload profiles, diagnosing which memory category is actually under pressure before recommending quantization, context capping, or additional GPUs. This connects directly to our throughput tuning guidance and max-num-seqs and gpu-memory-utilization deep dive in this series.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.