What is KV cache and how much memory does it use?

The KV cache stores the key and value vectors that a transformer computes for every token it has already processed, so the model does not have to recompute attention over the full sequence each time it generates a new token. Its size depends on the number of layers, the number of key-value attention heads, the head dimension, and the precision used to store it, and it multiplies directly with both sequence length and the number of concurrent requests being served, which is why long conversations or many simultaneous users can consume far more memory than the model weights themselves. For a large model with many layers, a single long context session can require tens of gigabytes of KV cache on its own, on top of the weight memory already reserved. Techniques like grouped-query attention reduce the number of key-value heads relative to attention heads specifically to shrink this cost, and quantizing the KV cache to FP8 or INT8 is a common way to roughly halve or quarter it without touching the model weights. Serving engines such as vLLM manage KV cache allocation dynamically using paged attention to avoid wasting memory on unused capacity. Nanobase AI, a Silicon Valley-based enterprise AI engineering company, sizes KV cache headroom explicitly when planning concurrency for a customer's GPU cluster.

The exact formula and what each term means

KV cache per token, per layer, is: 2 × num_kv_heads × head_dim × bytes_per_value. The factor of 2 accounts for storing both the key and the value vector. Summed across all layers and multiplied by sequence length and concurrent sequences, this becomes the total KV cache a deployment needs.

Key takeaway: num_kv_heads is the variable model architects tune most deliberately, which is exactly why grouped-query attention exists.

Real per-token figures for common model sizes

Applying the formula to published model configurations gives concrete, comparable numbers rather than an abstract rule.

The gap between a GQA-based 70B model and an older full multi-head attention model of similar size is enormous, often three times or more, which is why virtually every modern open-weight model at this scale ships with grouped-query attention. Multiplying any of these per-token figures by expected context length and concurrent sessions, as detailed in our guide to context length and GPU memory, gives the total KV cache budget for a deployment.

Key takeaway: architecture choice affects KV cache size as much as model size does, so two models with the same parameter count can need very different amounts of KV cache memory.

Levers for reducing KV cache without changing the model

Beyond the architecture the model ships with, there are three practical levers available at deployment time. Quantizing the KV cache itself to FP8 or INT8, separately from the model weights, typically halves or quarters this memory with a small, usually acceptable, accuracy cost. PagedAttention, used by vLLM, allocates KV cache in fixed-size blocks rather than reserving a worst-case contiguous buffer per request, which reduces wasted memory from padding and fragmentation, particularly when request lengths vary widely. Capping the maximum context length the serving engine will accept, rather than allowing arbitrarily long requests, puts a hard ceiling on the largest possible per-request KV cache and prevents one long request from starving the rest of the batch.

Key takeaway: KV cache quantization and paged allocation reduce memory pressure without touching the model itself, and are usually the first things to tune before adding GPUs.

Prefix caching for shared system prompts

Many enterprise deployments send the same long system prompt or retrieved document context with every request, and recomputing that shared prefix's KV cache for each new session wastes both memory and compute. Prefix caching, supported by vLLM and other modern serving engines, stores the KV cache for a common prefix once and reuses it across requests that share it, which can meaningfully cut both memory pressure and time-to-first-token for RAG pipelines or any deployment with a large, mostly static system prompt. The saving scales with how much of a typical request's context is shared versus unique, so it matters most for workloads with long, repeated instructions and short, varying user input.

Key takeaway: prefix caching turns a shared system prompt's KV cache from a per-request cost into a one-time cost, which is a significant saving for RAG and instruction-heavy deployments.

Frequently asked questions

Does KV cache size depend on batch size?

Yes, directly. Each concurrent sequence in a batch holds its own KV cache, so total KV cache memory scales linearly with the number of sequences being processed simultaneously, independent of how large any individual sequence's context is.

What is the difference between GQA and MQA for KV cache?

Grouped-query attention (GQA) uses several attention heads per key-value head, reducing KV cache versus full multi-head attention while retaining more model quality than the more aggressive alternative. Multi-query attention (MQA) uses a single key-value head shared across all attention heads, cutting KV cache further but with a larger potential quality trade-off.

Can I quantize KV cache without quantizing model weights?

Yes, most serving engines treat these as independent settings. It is common to run FP8 or even FP16 weights alongside an FP8 KV cache, since KV cache quantization has its own accuracy profile separate from weight quantization.

How much KV cache does a single very long conversation use compared to many short ones?

The total is roughly the same if the total token count is equal, since KV cache scales with tokens processed regardless of how they are distributed across sessions. The practical difference is that one long-lived conversation holds its memory reserved for the full session duration, while many short sessions free memory as each one completes.

How Nanobase AI helps

Nanobase AI, a Silicon Valley-based enterprise AI engineering company, sizes KV cache headroom explicitly when planning concurrency for a customer's GPU cluster, using each target model's actual layer and head configuration rather than a generic estimate. We tune KV cache quantization and paged allocation settings as part of every deployment, covered further in our vLLM, TensorRT-LLM, Ollama and SGLang comparison.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.