How many H100s are needed to serve a 405B model?

Serving a 405 billion parameter model like Llama 3.1 405B needs about 810 GB of VRAM in FP16, roughly 405 GB in FP8, and about 203 GB in INT4 for the weights alone, which rules out fewer than several H100 GPUs regardless of precision. FP8, the precision most commonly used in production for this model size, needs at least six 80 GB H100 GPUs just for the weights, but the practical minimum in most deployments is a full 8x H100 node providing 640 GB, since that leaves roughly 235 GB of headroom for KV cache, activation memory and concurrent requests rather than running the weights alone with no margin. INT4 quantization could theoretically fit on a 4x H100 configuration, though with meaningfully more accuracy trade-off at this model size than smaller models typically show, and it is less common in production for a flagship dense model like this. Tensor parallelism across all eight GPUs is standard for FP8 deployment, and NVLink bandwidth within the node becomes a real factor in throughput at this scale. An H200-based node offers more comfortable headroom for the same GPU count. Nanobase AI, a Silicon Valley enterprise AI engineering company, has sized and deployed 405B-class models on both H100 and H200 8-GPU nodes for enterprise customers.

Why "minimum that fits" and "minimum that's usable" diverge sharply at this scale

At 405 billion parameters, the gap between the theoretical minimum GPU count and a production-viable one is wider than for smaller models, because there is almost no slack left over once the weights are loaded on a bare-minimum configuration. Six 80 GB H100 GPUs provide exactly 480 GB, just above the roughly 405 GB FP8 weight footprint, but that leaves next to nothing for KV cache, activation buffers or CUDA overhead. A configuration sized to the weights alone, with no meaningful headroom, is not a production configuration; it is a demonstration that the model loads.

Comparing realistic node configurations

The full 8x H100 node is the practical minimum most deployments converge on, not because six GPUs cannot technically hold the weights, but because the resulting headroom at six is too thin for any meaningful concurrency or context length once real traffic arrives.

What the interconnect adds to this decision

Tensor parallelism across all eight GPUs in a node is standard for FP8 deployment of a model this size, splitting each layer's computation across the full GPU set rather than only splitting which layers live where. This makes NVLink bandwidth within the node a real factor in throughput, not just memory, since every forward pass requires frequent communication between GPUs to reconcile the split computation. A node with weaker inter-GPU bandwidth will show lower tokens-per-second even with sufficient combined memory, which is a separate consideration from the memory sizing table above and worth validating through benchmarking rather than assuming from GPU count alone.

Decision points for H100 versus H200 at this scale

1. Budget already committed to H100 infrastructure: an 8x H100 node is workable for FP8 serving with real, if not generous, headroom.
2. Planning for growth beyond a single 405B model, longer context, higher concurrency, or additional models on the same cluster: an H200-based node's roughly 700+ GB of headroom absorbs that growth without a hardware change.
3. Considering INT4 to reduce GPU count: technically feasible on four H100s, but 405B-class dense models show a more noticeable accuracy trade-off at this aggressive a quantization level than smaller models typically do, making it less common in production for a flagship model like this.
4. Evaluating a mixture-of-experts alternative instead of a 405B dense model: models like DeepSeek can offer comparable or better quality with different memory-versus-compute trade-offs, covered in MoE memory versus active parameters.

For most enterprise deployments of a 405B-class model, an 8x H100 or 8x H200 node, not a bare-minimum GPU count, is the configuration that actually reaches production.

Frequently asked questions

Is INT4 quantization common for 405B-class models in production?

Less common than for smaller models, since the accuracy trade-off at this scale tends to be more noticeable and the GPU savings, four H100s instead of eight, matters less once an 8-GPU node is already standard enterprise infrastructure for models of this size.

Does an 8x H100 node have enough headroom for high concurrency?

Roughly 220 to 230 GB of headroom after FP8 weights supports meaningful concurrency and moderate context length, though very high concurrency or long-context workloads should be load-tested rather than assumed, since KV cache demand scales quickly as shown in context length's impact on GPU memory.

Can a 405B model be split across two 4-GPU nodes instead of one 8-GPU node?

Technically possible with pipeline parallelism across nodes, but this introduces cross-node communication latency that a single 8-GPU node with NVLink avoids entirely, making a single node the simpler and generally faster choice when available.

Is H200 worth the added cost over H100 for a 405B deployment?

It depends on growth plans; H200 offers substantially more headroom for the same GPU count, which matters most if context length, concurrency or additional models are expected to grow, while a stable, single-model 405B deployment on already-owned H100 hardware may not need the upgrade yet.

How Nanobase AI helps

Nanobase AI, a Silicon Valley enterprise AI engineering company, has sized and deployed 405B-class models on both H100 and H200 8-GPU nodes for enterprise customers, validating tensor parallelism configuration and real throughput before finalizing a hardware recommendation. Learn more about the broader landscape in H100 vs H200 vs B200 for LLM inference.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.