A 70 billion parameter model needs about 140 GB of VRAM for its weights in FP16 or BF16, versus roughly 70 GB in FP8 and about 38 GB in INT4 formats such as AWQ or GPTQ. FP16 weights alone already exceed a single H100 or H200's memory, so a 70B model at full precision needs at least two 80 GB GPUs or one 141 GB H200 with almost no room left for KV cache, while INT4 shrinks the footprint enough to fit on a single 48 GB or 80 GB card with meaningful headroom for context and concurrent users. Add 20 to 50 percent on top of the weight figure for KV cache, activation memory and framework overhead, more at longer context lengths and higher batch sizes. The trade-off is that FP16 preserves the most accuracy while INT4 introduces a small but measurable quality loss on some tasks, so many enterprises land on FP8 as the practical middle ground on H100 or H200 hardware. Choosing between them depends on available GPUs, target concurrency and how sensitive the use case is to precision. Nanobase AI benchmarks both formats on customer workloads before recommending a quantization strategy and GPU count.
The full GPU-count table across precisions
The weight figures alone (140 GB FP16, 70 GB FP8, 38 GB INT4) only answer half the question, because GPU count depends on how much of that memory a single card holds and how much headroom is left for KV cache and concurrency. The table below assumes a moderate serving workload of 4K to 8K average context with continuous batching.
| Precision | Weights | Fits on | Practical GPU count for production headroom |
|---|---|---|---|
| FP16 / BF16 | ~140 GB | Nothing under 141 GB single-GPU | 1x H200, or 2x H100/A100 with tensor parallelism |
| FP8 | ~70 GB | 1x H100, H200 or A100 | 1 GPU minimal, 2 GPUs for real concurrency headroom |
| INT4 (AWQ/GPTQ) | ~38 GB | 1x L40S, RTX PRO 6000, or half an H100 | 1 GPU comfortable, including KV cache for moderate context |
Key takeaway: FP16 forces at least two data-center GPUs or one H200, while FP8 and INT4 both open the door to single-GPU serving with room to spare.
Working the KV cache math for a real deployment
Weight memory is fixed once you pick a precision, but KV cache is not, and for a 70B model it is large enough to change the GPU count on its own. Llama 3.3 70B uses grouped-query attention, keeping KV cache to roughly 320 KB per token in FP16, so a single 8K-token conversation costs about 2.5 GB, and ten concurrent 8K-token sessions cost about 25 GB, before weights are even counted. Stack that on 70 GB of FP8 weights and a single 80 GB H100 is already nearly full.
This is why FP8 weights plus FP8 KV cache, halving that 320 KB per token again, has become the default production configuration for 70B models: it buys back the margin FP16 KV cache would otherwise consume. Sizing correctly means computing this figure for your actual expected concurrency and context length, not just checking the weights fit.
Key takeaway: KV cache for a handful of concurrent 70B sessions can rival the weight memory itself, which is why FP8 weights plus FP8 KV cache is the common production default.
Where H200 and RTX PRO 6000 change the picture
An H100's 80 GB is the tightest fit at FP8, leaving roughly 10 GB after weights before the KV cache math above starts to bite. An H200's 141 GB leaves about 70 GB of headroom at the same weight size, comfortably absorbing dozens of concurrent 8K-token sessions on one GPU. A single RTX PRO 6000 at 96 GB sits between the two, and two of them (192 GB) hold even FP16 weights with headroom, at lower cost than an NVLink-connected H100 pair; specifics are in 70B on two RTX PRO 6000 GPUs.
Key takeaway: H200 buys single-GPU headroom at FP8, while a pair of RTX PRO 6000 cards is often the cheaper way to reach the same combined memory as two H100s.
Choosing between FP16, FP8 and INT4 for this model size
The decision is rarely about whether the model fits; INT4 always fits on modest hardware. It is about how much accuracy the workload can trade for headroom. FP16 is worth the extra GPUs mainly for evaluation baselines or heavy multi-step reasoning. FP8 is the practical default for production 70B serving on Hopper or Blackwell, since it is natively accelerated with a small quality delta. INT4 earns its place when hardware footprint is the binding constraint and the workload is chat or general Q&A rather than precise reasoning.
- Start with FP8 as the default unless you have a specific reason to avoid it.
- Move to INT4 only if hardware budget or footprint forces it, and validate output quality on your own prompts first.
- Reserve FP16 for cases where accuracy loss is unacceptable and the extra GPU cost is justified.
Key takeaway: FP8 is the right default for 70B production serving; INT4 is a hardware-driven choice that needs its own quality check, not a free upgrade.
Frequently asked questions
Does a 70B model in FP16 really need two GPUs?
Yes in almost every case. The 140 GB of FP16 weights exceeds the 80 GB on a single H100 or A100, so it needs tensor parallelism across two GPUs, or a single 141 GB H200 if you want to avoid the multi-GPU communication overhead while still running FP16.
How much smaller is INT4 than FP8 for a 70B model, in practice?
Roughly half. FP8 weights land around 70 GB, INT4 around 38 GB, so INT4 needs about 46 percent less memory than FP8 for the weights, which is often the difference between needing two cards and needing one.
Is FP8 accurate enough for a 70B model in production?
For most enterprise chat, summarization and retrieval-augmented use cases, yes. FP8 is natively supported on Hopper and Blackwell tensor cores with a small quality gap versus FP16, though validating on your own evaluation set before committing is worthwhile.
What happens if I underestimate KV cache for a 70B deployment?
Requests start queuing or failing with out-of-memory errors once concurrent sessions or context length exceed what the leftover VRAM after weights can hold, even though the model loaded successfully at startup. This is the most common cause of production 70B deployments that "worked in testing" but degrade under real traffic.
How Nanobase AI helps
Nanobase AI benchmarks FP16, FP8 and INT4 versions of a target 70B model against a customer's actual concurrency and context requirements before recommending GPU count, rather than sizing from the weight figure alone. That includes the KV cache math for real traffic, quantization quality validation on the customer's own prompts, and hardware selection across H100, H200, A100 and RTX PRO 6000. See how the same trade-offs play out at larger scale in sizing GPUs for 70B, 405B and DeepSeek R1.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.