How does batch size affect GPU memory and throughput?

Increasing batch size raises GPU memory usage because every request in the batch holds its own KV cache and activation buffers, so memory scales roughly linearly with the number of sequences processed simultaneously, while throughput generally improves with larger batches because the GPU's compute units are utilized more efficiently per token generated. Beyond a certain point, however, larger batches stop improving throughput proportionally and instead just consume more memory, and pushing batch size too far without enough headroom leads to out-of-memory errors or forces the serving engine to queue and delay requests. Continuous batching, used by vLLM and TensorRT-LLM, manages this trade-off dynamically by adding and removing requests from an active batch as they arrive and finish, which improves both memory efficiency and throughput compared to fixed, static batching. Latency for an individual request can also increase at very large batch sizes, since more work competes for the same compute resources, so there is a practical ceiling that depends on the acceptable time-to-first-token for a given use case. Finding the right batch size setting is workload-specific and requires testing rather than assuming a default value is optimal. Nanobase AI tunes batch size and memory utilization settings as part of every deployment it configures.

The trade-off curve in practice

Memory and throughput both rise with batch size, but not at the same rate or for the same duration, which is what creates a practical sweet spot rather than an "always bigger is better" relationship.

Key takeaway: throughput gains from increasing batch size flatten well before memory runs out, so the highest batch size a GPU can technically hold is rarely the best one to run.

Static batching versus continuous batching

Static batching groups a fixed set of requests together and waits for all of them to finish before starting the next batch, which wastes GPU capacity whenever requests in the batch finish generating at different times, since faster-finishing requests leave their slot idle rather than being replaced. Continuous batching, used by vLLM and TensorRT-LLM, admits new requests into an active batch as soon as any slot frees up, keeping the GPU's compute units consistently busy and improving both memory efficiency and aggregate throughput compared to the static approach. This difference is one of the single largest levers in real-world serving efficiency, often mattering more than the choice of GPU itself for chat-style workloads with variable response lengths.

Key takeaway: continuous batching alone often delivers a larger throughput improvement than upgrading to a bigger GPU, for workloads with variable-length responses.

Tuning batch-related settings in practice

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 64 \
  --gpu-memory-utilization 0.90

--max-num-seqs caps how many sequences can be processed concurrently, directly bounding the memory continuous batching will consume. --gpu-memory-utilization sets how much of the GPU's memory the engine is allowed to claim for weights and KV cache combined, leaving the remainder as a safety margin for activation spikes. Neither setting has a universally correct value; both need to be tuned against your specific model, hardware and traffic pattern through load testing, since setting them too conservatively wastes capacity and setting them too aggressively risks instability under peak load.

Key takeaway: max-num-seqs and gpu-memory-utilization are the two settings that most directly translate the batch size trade-off into a stable, tuned production configuration.

Why batching affects prefill and decode differently

Batch size does not act uniformly across a request's lifecycle. Prefill, processing the input prompt, is largely compute-bound, so batching several prefill operations together improves GPU utilization similarly to batching decode steps. Decode, generating each output token one at a time, is memory-bandwidth-bound, and batching multiple concurrent decode steps together is precisely what lets a GPU stream weights from memory once and apply them to many sequences, which is the core efficiency gain continuous batching is built around. Mixing prefill and decode work in the same batch, as continuous batching engines do, is what keeps both phases contributing to throughput rather than one phase starving the other during high-concurrency serving.

Key takeaway: continuous batching's real advantage is mixing compute-bound prefill with memory-bound decode in the same batch, which keeps both phases efficient at once.

Frequently asked questions

Does a larger batch size always mean higher latency for every request?

Generally yes for average latency, since more requests compete for the same compute at once, but continuous batching mitigates the worst effects by managing which requests are actively being processed rather than forcing every request to wait for an entire fixed-size batch to complete.

How do I know if my batch size setting is too conservative?

If GPU utilization stays well below its ceiling during load testing while requests queue rather than being admitted, the batch size or memory utilization setting is likely too conservative and can be raised to use available capacity more fully.

What causes out-of-memory errors even when average batch size looks fine?

Spikes in concurrent request volume or a cluster of unusually long requests arriving together can push actual memory usage above what an average-case configuration assumed, which is why load testing needs to include burst scenarios, not just steady-state averages.

Is batch size tuning different for chat versus batch/offline workloads?

Yes, offline or batch-processing workloads, where latency per individual request matters less, can generally push batch size higher for maximum aggregate throughput, while interactive chat workloads need to balance throughput against acceptable per-request latency more carefully.

How Nanobase AI helps

Nanobase AI tunes batch size, max-num-seqs and memory utilization settings as part of every deployment it configures, validating the result against realistic load including burst traffic rather than steady-state averages alone. This tuning work is covered in more depth in our comparison of vLLM, TensorRT-LLM, Ollama and SGLang. See our solutions page for the full deployment and tuning service.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.