PagedAttention is the memory management technique behind vLLM that stores each sequence's key-value cache in fixed-size, non-contiguous blocks, similar to how an operating system manages virtual memory pages, instead of requiring one large contiguous memory allocation per request. It matters because KV cache is the dominant memory cost of LLM serving at scale, and prior systems had to over-allocate memory for the worst-case sequence length, which fragmented GPU memory and left much of it unusable, sometimes wasting sixty to eighty percent of KV cache memory in naive implementations. By allocating cache in small blocks on demand and mapping them through a block table, PagedAttention eliminates that fragmentation and internal waste, which directly increases how many concurrent sequences fit on a GPU at once. More concurrent sequences means higher achievable batch sizes for continuous batching and therefore significantly higher throughput per GPU for the same hardware. PagedAttention also enables efficient memory sharing between sequences that share a prefix, which is the mechanism prefix caching builds on. It is the core reason vLLM became the reference implementation many other engines compare themselves against. Nanobase AI relies on PagedAttention-based engines as the default for customer deployments that need high concurrency per GPU.
The problem it replaced, described precisely
Before PagedAttention, serving engines allocated each sequence's key-value cache as one contiguous block of GPU memory, sized for the maximum sequence length the system supported, reserved at the start of generation. This is the direct cause of two separate kinds of waste. Internal fragmentation happens when a sequence generates far fewer tokens than the maximum reserved, leaving allocated memory unused for that sequence's entire lifetime. External fragmentation happens when sequences of different lengths finish and free memory at different times, leaving gaps too small and scattered to fit a new sequence's contiguous allocation, even when total free memory would be enough.
Both problems stem from the same root cause: treating KV cache as one contiguous allocation per sequence, which is exactly the assumption PagedAttention removes.
How the block table mechanism actually works
PagedAttention divides KV cache into fixed-size blocks, each holding the key and value vectors for a small, fixed number of tokens. A sequence's cache is then a list of block pointers, called a block table, mapping logical token positions to physical blocks that can live anywhere in GPU memory, not necessarily adjacent to each other. When a sequence needs more cache space, the system allocates one more block from a shared pool and adds a pointer to the block table; it does not need a contiguous region large enough for the sequence's eventual full length, because it never allocates that eagerly in the first place.
This is deliberately analogous to how an operating system's virtual memory manages pages: a process's memory looks contiguous to the process itself, while the physical pages backing it can be scattered anywhere in RAM.
Block size is a real tuning knob, not an implementation detail
| Block size | Effect on fragmentation | Effect on overhead |
|---|---|---|
| Smaller blocks | Less internal fragmentation, cache allocated closer to exact need | More block table entries to track and manage per sequence |
| Larger blocks | More potential internal fragmentation within the last partial block | Fewer block table entries, slightly less bookkeeping overhead |
Most engines default to a block size that balances these two costs reasonably well across typical workloads, but very long-context or very short-request workloads can benefit from revisiting the default rather than assuming it is universally optimal.
What this unlocks beyond just fixing fragmentation
Because sequences no longer need physically contiguous cache, PagedAttention also enables efficient copy-on-write sharing between sequences that share a prefix: two sequences with an identical system prompt can point their block tables at the same physical blocks for that shared portion, only diverging into separate blocks once their content differs. This shared-block mechanism is the foundation that automatic prefix caching builds on, and it is also what makes techniques like parallel sampling, generating several candidate outputs from one prompt, memory-efficient rather than requiring a full cache copy per candidate.
Why it became the reference point for the whole field
The core insight, reduce KV cache waste by removing the contiguous-allocation constraint, was general enough that essentially every serving engine that followed adopted some version of block-based or tree-based KV cache management, including SGLang's RadixAttention, which extends the same block-based idea into a tree structure for non-exact prefix matches. Understanding PagedAttention is therefore useful even when evaluating engines that do not use vLLM's exact implementation, since the tradeoffs it introduced, block size, table overhead, and shared-block eviction policy, recur across the field. The direct downstream benefit for prompt reuse is covered in what prefix caching is and how much it helps.
Frequently asked questions
Does PagedAttention change the model's mathematical output?
No, it only changes how KV cache memory is physically laid out and managed; the attention computation itself is mathematically identical to a contiguous-cache implementation, so output quality is unaffected.
Is PagedAttention specific to vLLM only?
The technique originated in and is most associated with vLLM, but the underlying idea of block-based, non-contiguous KV cache management has influenced or been adopted in some form by other serving engines since its introduction.
How much more GPU memory does block table overhead use?
The block table itself is small relative to the KV cache blocks it points to, typically a negligible fraction of total memory; the memory savings from eliminating fragmentation far outweigh this bookkeeping cost in practice.
Can block size be changed without restarting the server?
No, block size is a startup configuration for the serving engine and is fixed for the life of that server process; changing it requires restarting with a new configuration, so it should be tuned based on expected workload before deployment, not adjusted live.
How Nanobase AI helps
Nanobase AI, a Silicon Valley applied AI engineering company, configures block-based KV cache management and related memory settings specifically for each customer's context length and concurrency profile, since default block sizes are not tuned for every workload shape. This is part of Nanobase AI's inference engineering work within its broader GPU infrastructure and LLM deployment practice.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.