Continuous batching is a scheduling technique that adds new requests into a running inference batch and removes finished ones at every generation step, rather than waiting for an entire fixed batch of requests to complete before starting the next one. Traditional static batching wastes GPU cycles because a batch only finishes when its slowest sequence finishes, so short requests sit idle waiting on long ones; continuous batching, sometimes called in-flight batching in TensorRT-LLM, instead treats each decoding step as an opportunity to reshuffle which sequences occupy the batch. This keeps GPU utilization consistently high under variable-length, variable-arrival-time traffic, which is exactly the pattern real chat and agent workloads produce. The practical effect is substantially higher throughput per GPU, commonly several times better than naive batching implementations, along with more consistent latency because new requests do not have to wait for an entire batch cycle to begin. vLLM, TensorRT-LLM, and SGLang all implement their own version of this technique, paired with efficient KV cache memory management like PagedAttention to make it work at scale. Nanobase AI configures continuous batching parameters specifically for each customer's request-length and concurrency profile rather than relying on defaults.
Watching one decoding step to see the difference
The clearest way to understand continuous batching is to watch a single decoding step under each approach. Under static batching, the engine collects a fixed group of requests, runs every one of them through the model together until the group is done, and only then admits a new batch. If nine of ten requests finish generating after fifty tokens and one runs to five hundred tokens, the GPU keeps processing that batch of ten, with nine slots doing nothing useful, until the last one finishes.
Under continuous batching, the scheduler re-evaluates the batch composition at every single decoding step. The moment one of those nine short requests finishes, its slot is freed and a new, waiting request is admitted immediately, without waiting for the long-running tenth request to complete.
That per-step re-evaluation is the entire mechanism; everything else, including throughput gains and better latency consistency, is a downstream consequence of scheduling at the granularity of one token generation step instead of one whole batch.
What each major engine calls the same idea
| Engine | Terminology | Notable scheduler detail |
|---|---|---|
| vLLM | Continuous batching | Integrated with PagedAttention's block-based KV cache allocation |
| TensorRT-LLM | In-flight batching | Built into the compiled engine's runtime scheduler |
| SGLang | Continuous batching | Combined with RadixAttention-aware scheduling for prefix reuse |
| Triton Inference Server | Dynamic batching (configurable) | Can layer on top of a backend's own batching, or defer to it |
The terminology differs but the underlying goal is identical: keep the GPU processing useful work at every step rather than idling on the slowest member of a fixed group.
How chunked prefill interacts with the scheduler
A complication continuous batching has to handle is that prefill, processing a new prompt's initial tokens, is compute-intensive and can take much longer than a single decode step for other in-flight sequences. Without special handling, a long incoming prompt could stall every other active request's next token while it prefills. Chunked prefill solves this by breaking a long prompt's prefill into smaller pieces interleaved with ongoing decode steps for other sequences, so no single new request can monopolize a scheduling round. This is why continuous batching and chunked prefill are usually discussed together in the same engine's scheduler documentation, even though they solve related but distinct problems.
Why this matters more as traffic becomes bursty and variable
Continuous batching's advantage grows with request variability. A workload with uniform prompt and output lengths sees a smaller gap between static and continuous batching, since there is little idle time to reclaim. Real chat and agent traffic is the opposite: prompt lengths vary by an order of magnitude, output lengths vary even more depending on whether a user asked a yes-or-no question or requested a long document, and requests arrive at unpredictable times rather than in neat batches. That variability is exactly the condition under which continuous batching's per-step scheduling produces meaningfully higher GPU utilization than any fixed-batch approach.
What to check when evaluating a serving engine's batching
- Confirm the engine implements true per-step batch admission, not just periodic batch refresh at a fixed interval, since the latter captures only part of the benefit.
- Check whether chunked prefill (or an equivalent mechanism) is supported, since without it a long prompt can still create a latency spike for other users.
- Verify how the engine handles preemption under memory pressure, since a scheduler needs a policy for what happens when the KV cache manager runs out of room for all active sequences.
- Test with your actual traffic's length variability, not a benchmark dataset with uniform prompts, since that variability is what continuous batching is designed to exploit.
This scheduling behavior is inseparable from KV cache management, covered in depth in what PagedAttention is and why it matters.
Frequently asked questions
Does continuous batching increase latency for any individual request?
It can add small, bounded scheduling overhead per step, but in exchange it usually reduces overall queueing delay dramatically, since new requests do not wait for an entire batch cycle to complete before starting, which is the dominant latency factor under static batching.
Is continuous batching only useful for high-traffic deployments?
The GPU utilization benefit is largest under high concurrency, but even moderate traffic with variable request lengths benefits, since static batching's idle-slot waste appears whenever request lengths differ, not only at extreme scale.
Does continuous batching change the actual model output?
No, it changes only the scheduling of requests through the model, not the model's computation or sampling behavior; a given request produces the same output regardless of what else is being batched alongside it.
Do all inference engines implement continuous batching the same way?
No, the core idea of per-step batch admission is shared, but implementation details differ, including how each engine handles chunked prefill, preemption policy, and KV cache eviction, which is why throughput can differ between engines even with continuous batching enabled in all of them.
How Nanobase AI helps
Nanobase AI tunes scheduler and batching configuration for each customer's specific request-length and concurrency profile rather than relying on default settings, since the throughput gain from continuous batching depends heavily on how well it is configured for the actual traffic pattern. This tuning is part of Nanobase AI's inference engineering and GPU infrastructure services.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.