How much VRAM is needed for 128K context on a 70B model?

A single 128,000-token session on a 70B model like Llama 3.3, which uses grouped-query attention with a reduced number of key-value heads, typically needs on the order of 40 GB of KV cache at FP16, on top of the roughly 140 GB of FP16 weights, so one 128K-context request alone can approach or exceed the 80 GB capacity of a single H100. That pushes a real 128K-context deployment toward an H200 with 141 GB, toward FP8 weights of about 70 GB paired with a quantized KV cache to stay well under 141 GB, or toward splitting the model across two GPUs with tensor parallelism to get combined memory well above what any single card offers. Every additional concurrent 128K session adds a comparable amount of KV cache, so serving more than one or two long-context users at once on a single GPU is rarely realistic without quantization or multiple GPUs. Models with fewer key-value heads or newer attention variants reduce this cost meaningfully compared to older architectures without grouped-query attention. Nanobase AI stress-tests long-context memory usage on real hardware rather than relying on theoretical figures before sizing a cluster.

Walking through the calculation

Rather than quoting the 40 GB figure on its own, it is worth deriving it, since the same steps apply to any model and context length you need to size.

Key takeaway: one 128K-token session on a 70B model already needs more memory than an H100 provides once weights are included, making the H200 the natural single-GPU answer.

What changes with a non-GQA model or a second session

The 41 GB figure assumes grouped-query attention with a reduced key-value head count; a model without GQA can need three times or more KV cache for the same context length, which pushes the single-session total well past even an H200's capacity. Similarly, a second concurrent 128K session roughly doubles the KV cache portion, from 41 GB to about 82 GB, which combined with 70 GB of FP8 weights exceeds 141 GB and forces a move to two GPUs regardless of which model architecture is in use.

Key takeaway: every additional concurrent 128K session or every step up in weight precision pushes a 70B long-context deployment further past single-GPU capacity. The concurrency side of this trade-off is covered further in concurrent users per H100.

Reducing the footprint without losing the context window

Quantizing the KV cache itself to FP8, independent of weight precision, roughly halves the 41 GB figure to about 20 GB, which meaningfully changes the picture: 70 GB FP8 weights plus 20 GB FP8 KV cache totals about 90 GB, still slightly over an H100 but comfortably inside an H200 with real headroom left for a second session or moderate additional concurrency. This combination, FP8 weights with FP8 KV cache, is the standard configuration for teams that need to support 128K-class context on a single GPU rather than committing to tensor parallelism across two.

Key takeaway: FP8 KV cache quantization is often what makes 128K context on a 70B model fit a single H200 rather than requiring two GPUs.

Sustained versus peak: what a 128K session costs over time

A single 128K-token session does not hold its full 41 GB of KV cache from the first token; the cache grows as the conversation or document processing proceeds, so early turns cost far less than the theoretical maximum. This matters for capacity planning because a pool of GPUs sized for several concurrent 128K sessions rarely has all of them simultaneously at their peak token count, which is why serving engines using dynamic, paged KV cache allocation can support somewhat more concurrent long-context sessions in practice than a naive worst-case sum would suggest, provided the deployment can tolerate occasional queuing when several sessions do peak together.

Key takeaway: KV cache for long sessions accumulates gradually, so dynamic allocation lets a GPU pool serve more concurrent 128K-class sessions than a strict worst-case calculation implies.

Frequently asked questions

Is 128K context realistic for a production chatbot, or mostly a document-analysis feature?

Most chat use cases rarely approach 128K tokens in a single session; it is far more common for document analysis, long-form research assistance, or codebase-wide coding tasks. Sizing for 128K should reflect the actual fraction of traffic expected to use it, not assume every session will.

Does a smaller model make 128K context proportionally cheaper?

Yes, KV cache per token scales with the model's layer count and key-value head configuration, both of which are smaller in smaller models, so a 7B or 13B model at 128K context needs a small fraction of the memory a 70B model requires for the same context length.

Can two H100s handle 128K context on a 70B model as well as one H200?

For a single session, yes, since two H100s provide 160 GB combined, more than the roughly 111 GB needed. For multiple concurrent 128K sessions, the two H100s' combined memory and the H200's larger single-GPU pool both need to be checked against the actual concurrency target, since neither is unlimited.

What is the simplest way to reduce memory pressure if 128K context is only occasionally needed?

Set a lower default max-model-len for most traffic and route the rare long-context requests to a separate pool of GPUs provisioned specifically for that case, rather than sizing every GPU in the fleet for the worst-case context length.

How Nanobase AI helps

Nanobase AI stress-tests long-context memory usage on real hardware, running the actual KV cache math for a customer's model and target context length rather than relying on theoretical figures, and recommends between single-GPU FP8 configurations and multi-GPU tensor-parallel setups based on measured results. Our H100 vs H200 vs B200 guide covers this trade-off across the full GPU lineup.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.