FP8 KV cache means storing the key-value cache, the memory that holds attention context for every token in every active sequence, in eight-bit floating point format instead of the default sixteen-bit BF16 or FP16, which roughly halves the memory footprint of the single largest consumer of GPU memory in high-concurrency LLM serving. You should generally enable it on Hopper, Blackwell, or newer GPUs like the H100, H200, and B200, which have native FP8 tensor core support, since the memory savings translate directly into either larger batch sizes, longer supported context lengths, or both, for the same GPU, often the single highest-leverage memory optimization available after model quantization itself. The accuracy impact is typically minimal for most workloads, since KV cache values have a narrower dynamic range than weights, but it is worth validating output quality on your own evaluation set before rolling it out broadly, particularly for tasks sensitive to long-context recall or precise numerical reasoning. Enabling it in vLLM is a single flag, kv-cache-dtype set to fp8, with no changes needed to the model checkpoint itself. Combined with FP8 model weights, this is one of the more effective ways to raise concurrency on the same GPU footprint. Nanobase AI enables and validates FP8 KV cache as a standard step in its inference tuning process.
Why KV cache tolerates aggressive quantization well
Model weight quantization and KV cache quantization are solving the same general problem, representing numbers with fewer bits, but they do not carry the same risk. Weights encode everything the model learned during training, and aggressive quantization can distort that in ways that show up as degraded reasoning or factual accuracy. KV cache values, by contrast, represent attention keys and values for already-processed tokens, and empirically these values have a narrower dynamic range than weights, which is why they tolerate FP8's reduced precision with less measurable quality impact than weight quantization at the same bit width typically shows.
This is not a guarantee that applies universally to every model and task, but it explains why FP8 KV cache is generally considered lower-risk than, say, INT4 weight quantization, and why it is often enabled before more aggressive weight compression is considered.
KV cache values have a narrower dynamic range than weights, which is the technical reason FP8 KV cache is lower-risk than equivalent weight quantization.
What it actually costs to enable
Turning on FP8 KV cache in vLLM is a single startup flag, kv-cache-dtype fp8, with no changes needed to the model checkpoint itself, since it quantizes the cache at runtime rather than requiring a pre-quantized model. This is meaningfully simpler than weight quantization, which typically requires either a pre-quantized checkpoint (AWQ, GPTQ) or a calibration step. The scaling approach vLLM uses is calibration-free for KV cache in most configurations, which further lowers the operational cost of trying it.
FP8 KV cache is one of the lowest-effort optimizations available in vLLM, a single flag with no checkpoint changes required, which makes it worth testing early in any tuning pass.
Comparing it to other cache-reduction techniques
| Technique | What it reduces | Effort to enable | Accuracy risk |
|---|---|---|---|
| FP8 KV cache | Bytes per cached token | Single flag, no checkpoint change | Low on modern hardware |
| Grouped-query attention (GQA/MQA) | Number of KV heads cached | Architectural, fixed at model training time | None at inference (built into the model) |
| Prefix caching | Redundant computation across requests | Single flag | None (exact reuse, not approximation) |
| Shorter max-model-len | Cache reserved per sequence | Configuration change | None (limits context, not precision) |
| Cache eviction / sliding window | Cache retained for old tokens | Architectural or engine-level | Can lose long-range context |
FP8 KV cache and prefix caching solve different problems and stack well together: prefix caching avoids recomputing shared context at all, while FP8 makes whatever cache does get stored smaller. Grouped-query attention is a model architecture decision made before you ever see the checkpoint, not something you configure at serving time, but it is worth knowing about since a model already using GQA has a smaller baseline KV cache to begin with, on top of whatever FP8 saves further.
FP8 KV cache and prefix caching address different sources of memory and compute cost respectively and are not competing options, they compound.
Where to validate before rolling out broadly
The accuracy impact is typically minimal, but "typically" is doing real work in that sentence, and validation on your own evaluation set is worth the modest effort before enabling it broadly in production. Tasks especially worth checking closely include long-context recall (does the model still retrieve information from early in a long conversation correctly) and precise numerical or structured reasoning, since these are the areas where reduced-precision cache is most likely to show any measurable effect if it shows one at all.
A practical rollout: enable FP8 KV cache in a staging environment, run your existing evaluation suite or a representative sample of production traffic through both FP16 and FP8 cache configurations, and compare outputs before enabling it in production.
Validate on your own long-context and precision-sensitive tasks before broad rollout, since "minimal impact" is a general observation, not a guarantee for every workload.
Frequently asked questions
Does FP8 KV cache require FP8 model weights as well?
No, they are independent settings; you can run FP8 KV cache with FP16, AWQ, or FP8 model weights. Combining FP8 weights and FP8 KV cache together typically produces the largest combined memory saving and the highest achievable concurrency for a given GPU.
Which GPUs support FP8 KV cache well?
Hopper (H100, H200) and Blackwell (B200) GPUs have native FP8 tensor core support, which is where this feature delivers its full benefit; older architectures like A100 lack native FP8 support and see less advantage.
How much does FP8 KV cache actually reduce memory by?
It roughly halves the memory footprint of the KV cache specifically, since it moves from 16-bit to 8-bit storage per cached value; the impact on total GPU memory usage depends on what fraction of memory the KV cache represents for your specific model and workload.
Can FP8 KV cache be combined with tensor parallelism?
Yes, the two are independent; tensor parallelism splits both weights and KV cache across GPUs, and FP8 KV cache reduces the size of whatever cache portion lives on each GPU after that split.
How Nanobase AI helps
Nanobase AI enables and validates FP8 KV cache as a standard step in its inference tuning process, checking long-context and precision-sensitive tasks on the customer's own evaluation data before recommending broad rollout. See our related guidance on reducing vLLM GPU memory usage for how this fits into the full memory optimization picture, and our H100 vs H200 vs B200 comparison for hardware-specific FP8 support details.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.