DeepSeek V3 generally does not run comfortably on a single 8x H100 server at its native FP8 precision, because its roughly 671 billion total parameters need about 670 GB of memory, which is tighter than the 640 GB of raw HBM an 8x H100 node provides once KV cache, activation buffers and CUDA overhead are accounted for. In practice, teams either move to an 8x H200 node, offering 1,128 GB and comfortable headroom, or quantize DeepSeek V3 down to INT4, which brings the weights to roughly 340 to 380 GB and fits well within a single 8x H100 server with room left for concurrent sessions. The mixture-of-experts architecture means the full parameter count has to be resident in memory even though only about 37 billion parameters activate per token, so there is no way to shrink the footprint by relying on active parameters alone. Serving frameworks such as vLLM and SGLang both support DeepSeek V3's architecture with expert parallelism across nodes. The right choice depends on whether the workload can tolerate INT4's small accuracy trade-off. Nanobase AI benchmarks both the FP8 and INT4 paths on customer hardware before recommending a deployment configuration.
The memory budget that decides the answer
Whether DeepSeek V3 fits on 8x H100 comes down to arithmetic, not a general rule, so it is worth laying out the actual numbers rather than treating it as fits-or-doesn't.
| Item | Memory |
|---|---|
| Raw HBM on 8x H100 | 640 GB |
| DeepSeek V3 FP8 weights (671B total params) | ~670 GB |
| CUDA context and framework overhead | ~10-15 GB |
| KV cache for any concurrent request | Additional, on top of the above |
| Resulting gap | Weights alone already exceed raw HBM by ~30 GB |
Key takeaway: FP8 weights alone are larger than the raw memory an 8x H100 node provides, before any KV cache is counted, so the native configuration does not fit.
Three ways to close the gap
Because the shortfall is specific and quantifiable, there are three concrete paths, each with a different trade-off, rather than a single correct answer.
- Move to 8x H200 (1,128 GB). This is the simplest fix: the weights fit with roughly 450 GB left over for KV cache and concurrency, and no quantization trade-off is introduced. The cost is the H200's premium over H100 hardware.
- Quantize to INT4. Bringing weights down to roughly 340 to 380 GB fits comfortably within 640 GB with meaningful headroom left for KV cache, at the cost of an accuracy trade-off that should be validated against your own evaluation set before production use.
- Split across two H100 nodes. Tensor and pipeline parallelism across two 8x H100 servers, connected by InfiniBand, provides 1,280 GB of combined memory, more than enough for FP8 weights and KV cache, at the cost of cross-node communication latency and double the server count.
Key takeaway: H200 avoids quantization but costs more per GPU, INT4 keeps you on H100 but trades some accuracy, and two H100 nodes keep FP8 but add network complexity.
Why the active-parameter count does not help here
DeepSeek V3 activates only about 37 billion of its roughly 671 billion parameters per token, and it is tempting to assume that means memory requirements are closer to a 37B dense model. They are not. A mixture-of-experts router can direct any token to any expert, so every expert has to be loaded and ready in GPU memory at all times, regardless of how few are used on a given forward pass. Memory footprint tracks total parameters; only compute cost per token tracks active parameters. This is the reason DeepSeek V3 needs as much memory as a 671B dense model would, while running with the compute profile of a much smaller one, which is also what makes it fast once it fits.
Key takeaway: size DeepSeek V3's memory footprint from its 671B total parameters, never from the 37B active-parameter figure.
Matching the path to the team's constraints
Which of the three paths makes sense depends less on the model and more on what a team already has and is willing to trade. A team with capital budget but no appetite for an accuracy review leans toward 8x H200. A team already standardized on H100 fleets, with time to validate output quality, leans toward INT4. A team that already operates multi-node InfiniBand clusters, for example from training workloads, often finds the two-node FP8 path the least disruptive, since it reuses existing networking rather than introducing a new hardware tier or a quantization validation process.
Key takeaway: the right path is usually the one that reuses hardware and processes a team already has, not necessarily the theoretically cleanest option.
Frequently asked questions
Why doesn't NVIDIA just ship an 8x H100 node with more memory?
The H100's 80 GB per GPU is fixed by its HBM3 stack design; it is not a configurable option. The H200, released as a memory and bandwidth upgrade on the same Hopper compute, exists specifically to address workloads like this where H100's 80 GB per GPU is the limiting factor rather than compute.
Is INT4 DeepSeek V3 good enough for production?
For many general-purpose tasks the accuracy loss is modest, but DeepSeek V3 is often used for tasks involving code and structured reasoning where quantization sensitivity varies by task. Test INT4 output against FP8 on your own representative prompts before deciding.
Does splitting DeepSeek V3 across two nodes hurt latency?
Yes, cross-node communication over InfiniBand is slower than intra-node NVLink, so a two-node deployment typically has higher latency per token than a single well-provisioned node, though it remains far faster than CPU offloading or excessive quantization on constrained hardware.
Can DeepSeek V3 and DeepSeek R1 share the same GPU sizing plan?
Largely yes, since both are 671B-parameter mixture-of-experts models with similar FP8 memory footprints, roughly 670 GB. The main difference is in behavior and output style rather than hardware requirements, so the same 8x H200, INT4-on-H100, or two-node strategies apply to both.
How Nanobase AI helps
Nanobase AI benchmarks both the H200 single-node path and the INT4-on-H100 path for DeepSeek V3 on customer hardware, measuring actual throughput and output quality before recommending a configuration, rather than relying on the weight arithmetic alone. Our guide to GPU sizing for 70B, 405B and DeepSeek R1 covers the same trade-offs across model sizes. See our solutions for full deployment support.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.