How many GPUs do you need for 70B, 405B and DeepSeek R1?

For production inference with headroom, a 70B model needs 2× H100 80 GB in FP8 (or 1× H200 141 GB), a 405B model needs 8× H100 in FP8 (or 4× H200), and DeepSeek R1 (671B MoE, released in FP8) needs one 8× H200 node or 16× H100. Each number comes from parameters × bytes per weight, plus 20–50% headroom for KV cache and runtime overhead, rounded up to a tensor-parallel degree of 1, 2, 4 or 8. Context length and concurrent users set the headroom, so the same 70B model can need 2 GPUs for a short-context chatbot and 8 for a 128k-token document pipeline.

The memory formula: weights, KV cache and overhead

Every sizing question reduces to one inequality: the weights, the KV cache for all tokens in flight, and the runtime's working memory must fit in the combined VRAM of the GPUs assigned to the model. Weights are fixed at load time, KV cache grows with every token served, and overhead is roughly constant per GPU.

GPU memory needed  =  weights + KV cache + overhead

weights   = parameters × bytes per weight
            FP16/BF16 = 2 bytes, FP8 = 1 byte, INT4 ≈ 0.55 bytes (incl. scales)
KV cache  = 2 × layers × KV heads × head_dim × bytes per value × tokens in flight
overhead  = CUDA context + activations + kernel workspace ≈ 5–10% of VRAM

For a 70B dense model that gives about 140 GB in FP16, 70 GB in FP8 and about 38 GB in INT4 (INT4 carries a few gigabytes of scale metadata). vLLM reserves a fixed fraction of VRAM (--gpu-memory-utilization, default 0.9) and hands everything left after weights to the KV cache. Hopper and Blackwell run FP8 natively, so FP8 halves memory and speeds up bandwidth-bound decode at almost no quality cost; INT4 (AWQ, GPTQ) halves it again with a small, model-dependent cost. If weights alone consume more than about 75–80% of total VRAM, there is no room for concurrent users: add GPUs or drop precision.

Sizing table: 7B to 671B on H100, H200, B200 and RTX PRO 6000

The first table gives weight memory only. The second gives the minimum GPU count with about 30% headroom for KV cache and overhead (the margin Nanobase AI uses in customer sizing), rounded up to a tensor-parallel degree of 1, 2, 4 or 8 inside a node and to multiples of 8 across nodes. Official capacities are 80 GB HBM3 for the H100, 141 GB HBM3e for the H200, about 180 GB HBM3e on B200 and 96 GB GDDR7 on RTX PRO 6000.

Minimum GPU count as FP16 / FP8 / INT4; "n/r" means not recommended (more than 8 PCIe GPUs without NVLink, or more than two nodes).

Three cells deserve a comment: 70B in FP16 loads on 2× H100 (140 of 160 GB) and on one B200, but leaves so little KV cache that only short-context, low-concurrency use works, hence 4 and 2. 405B in FP8 on 8× H100 (405 of 640 GB) is the classic single-node layout. DeepSeek R1 in native FP8 does not fit on 8× H100 (671 GB of weights); the answers are one 8× H200 node, 16× H100 across two nodes, or an INT4 checkpoint on 8× H100. When you can choose the GPU, buy the memory: one H200 or B200 replaces two H100s for 70B-class models, and an 8× H200 node is the smallest clean home for DeepSeek R1.

KV cache: how context length and concurrency change the answer

The KV cache stores the key and value tensors of every token in every active sequence so the model does not recompute them at each decode step. Its size per token depends on the architecture, not the prompt: grouped-query attention (GQA) models keep only 8 KV heads, and the multi-head latent attention (MLA) in DeepSeek V3 and R1 compresses the cache further.

What matters is tokens in flight across all concurrent sequences, not the configured maximum context, because paged attention in vLLM and TensorRT-LLM allocates KV blocks as tokens arrive: 20 users with 4k-token conversations cost 80k tokens of cache even if --max-model-len is 128k. FP8 KV cache (--kv-cache-dtype fp8) halves the footprint with negligible quality impact on most models, and prefix caching removes the cost of a shared system prompt.

Budget from the top down: usable VRAM (total × 0.9) minus weights minus a few gigabytes of overhead, divided by KV per token. For 70B in FP8 on 2× H100, 144 GB minus 70 GB minus about 6 GB leaves about 68 GB, roughly 200k tokens of FP16 KV cache: about 25 concurrent users at 8k tokens each, or 6 at 32k, and twice that with FP8 KV cache. Long context does not change the weights; it multiplies the concurrency you must fund with headroom, and it is the usual reason a 2-GPU plan becomes a 4-GPU plan.

MoE models: total parameters decide memory, active parameters decide speed

A mixture-of-experts model routes each token through a small subset of its feed-forward experts. DeepSeek V3 and R1 have 671B total parameters but activate about 37B per token (8 of 256 routed experts plus 1 shared expert in each of 61 layers); Llama 4 Maverick has 400B total and 17B active, Qwen3-235B-A22B 235B and 22B. Memory follows total parameters because any token can be routed to any expert, so every expert must be resident: about 671 GB for R1 in FP8 and about 400 GB for Maverick.

Speed follows active parameters, with a caveat. At batch size 1 a decode step reads roughly 37B parameters of R1, so per-token latency is closer to a 37B model than a 671B one; at large batch most experts are touched every step and bandwidth demand climbs toward the dense case. Expert parallelism (EP) spreads experts across GPUs and is how vLLM and SGLang scale MoE serving, though for one 8-GPU node plain TP=8 remains the simplest working setup as of 2026.

Do not confuse R1 with its distilled variants: R1-Distill-Llama-70B and R1-Distill-Qwen-32B are dense models and size like the 70B and 32B rows above. For MoE, budget memory on total parameters and expect latency close to active parameters; no configuration loads only the experts you expect to need.

Tensor and pipeline parallelism basics

Tensor parallelism (TP) slices every weight matrix across GPUs so each holds 1/TP of every layer. Every layer then needs an all-reduce, which is why TP lives inside a node on NVLink (900 GB/s per GPU on H100, 1.8 TB/s on B200); over PCIe, as with RTX PRO 6000, TP=2 or 4 works but loses throughput to interconnect waits. TP also aggregates bandwidth: TP=2 on H100 reads weights at a combined 6.7 TB/s, so a 70B model that fits on one H200 often decodes faster on two GPUs. TP must divide the attention heads evenly; Llama 70B has 64 query heads and 8 KV heads, so TP of 1, 2, 4 or 8 is clean and TP=16 wastes memory replicating KV heads.

Pipeline parallelism (PP) assigns contiguous groups of layers to different GPUs or nodes, and only activations cross a stage boundary. It needs far less bandwidth, which makes it the right tool across nodes on InfiniBand, but it adds pipeline bubbles at low batch sizes; a 16× H100 deployment of Llama 405B in FP16 is typically --tensor-parallel-size 8 --pipeline-parallel-size 2. Once a model fits, further GPUs are better spent on data-parallel replicas than on a wider TP group (engine differences are covered in vLLM vs TensorRT-LLM vs Ollama vs SGLang). Fit the model with the smallest TP that leaves 30% headroom, then scale throughput with replicas.

Worked example: sizing a 70B deployment for 200 users

Assume an on-premise deployment of Llama 3.3 70B for 200 internal users with a 16k-token context limit on Hopper GPUs (the surrounding stack is in the on-premise LLM deployment guide). The sizing runs in six steps.

1. Precision: FP8 is native on Hopper and near-lossless at 70B, so weights are about 70 GB.
2. Concurrency: plan for 10–15% of 200 users active at peak, so 20–30 concurrent sequences averaging 6k tokens in flight (4k prompt, 2k generation) with a worst case of 16k.
3. KV cache: at about 320 KB per token, 30 × 6k tokens is about 58 GB on average and 30 × 16k about 154 GB at worst; FP8 KV cache halves both to about 29 GB and 77 GB.
4. Overhead: the 10% vLLM reservation plus about 3 GB per GPU for activations and CUDA graphs.
5. Total: worst case with FP8 KV cache is about 70 + 77 + 10 = 157 GB. Two H100s (144 GB usable) cannot cover the peak; two H200s (about 254 GB usable) can with room to grow, and four H100s is the equivalent H100 layout.
6. Throughput check: two H200s read 70 GB of weights at a combined 9.6 TB/s, an upper bound of about 137 decode steps per second; real engines reach roughly 30–40% of that, so expect on the order of 40–55 tokens per second per user with 30 users active. This is a bandwidth estimate, not a benchmark; confirm it with a load test.

The resulting launch command for vLLM (flag details are in the vLLM documentation):

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 2 \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --max-model-len 16384 \
  --max-num-seqs 64 \
  --gpu-memory-utilization 0.90

The same procedure gives the other headline numbers. Llama 405B in FP8 under this load needs about 405 GB plus about 250 GB of FP16 KV cache, so 8× H100 covers the peak only with FP8 KV cache and 8× H200 is comfortable; DeepSeek R1 needs about 671 GB plus only about 35 GB of KV cache thanks to MLA, so on 8× H200 the model, not the cache, sets the size. Turning a GPU count into cost per token is covered in own GPUs vs cloud APIs. Size for the peak, not the average, and check that bandwidth-derived tokens per second meets your latency target before settling on a GPU count.

Frequently asked questions

Can I run a 70B model on a single GPU?

Yes, at reduced precision. In FP8 the 70 GB of weights fit one H200, B200 or RTX PRO 6000 96 GB, though the 96 GB card leaves only about 20 GB for KV cache; in INT4 (about 38 GB) one H100 80 GB works. FP16 weights are 140 GB and need two GPUs. Single-GPU 70B suits low concurrency and contexts up to about 16k tokens.

How much GPU memory does DeepSeek R1 need?

About 671 GB for the weights alone, because R1 is a 671B-parameter MoE released in FP8. The smallest standard deployment is one 8× H200 node (1,128 GB), which also leaves generous KV cache thanks to MLA. It does not fit on 8× H100 (640 GB) in FP8; use 16× H100 across two nodes or an INT4 checkpoint (about 350 GB) on 8× H100 or 4× H200.

Does FP8 or INT4 quantization hurt model quality?

FP8 is close to lossless for models of 8B and larger and is the default choice on Hopper and Blackwell. INT4 weight-only methods such as AWQ and GPTQ usually cost a small amount of accuracy, more on small models and multi-step reasoning tasks, and add a few percent of memory for scales. Run your own evaluation set against the quantized checkpoint before committing hardware to it.

How do I calculate KV cache size?

Multiply 2 (keys and values) × layers × KV heads × head dimension × bytes per value for the per-token cache, then multiply by all tokens in flight across concurrent requests. Llama 3.3 70B (80 layers, 8 KV heads, head dimension 128) costs about 320 KB per token in FP16, or about 10.7 GB for one 32k-token sequence. FP8 KV cache halves it; MLA models such as DeepSeek V3 need roughly a quarter.

Is the H200 worth it over the H100 for a 70B model?

Usually, yes. Its 141 GB lets a 70B FP8 model run on one GPU instead of two, and 4.8 TB/s of bandwidth (versus 3.35 TB/s) gives faster decode per GPU; two H200s serve 70B at 16k context for 30 concurrent users where the same load needs four H100s. If you already own H100s the gap does not justify replacement. See H100 vs H200 vs B200 for the full comparison.

Can I use RTX PRO 6000 cards instead of data-center GPUs?

Yes, for models up to about 70B and moderate concurrency. The 96 GB GDDR7 card holds a 70B FP8 model alone and a 405B INT4 model across four cards. The limits are no NVLink, which slows tensor parallelism over PCIe, and lower memory bandwidth (about 1.8 TB/s) than HBM parts, so per-user token rates are lower. It fits development, departmental deployments and workloads that do not need many GPUs per model.

How Nanobase AI can help

Nanobase AI sizes, installs and operates GPU infrastructure for private LLM deployments. We start from your models, context lengths and concurrency targets, produce a memory and throughput plan like the one above, and validate it with load tests on H100, H200, B200 or RTX PRO 6000 systems before you buy. We then deploy the serving stack (vLLM, TensorRT-LLM or NVIDIA NIM) with the right parallelism, Kubernetes GPU Operator or Slurm scheduling and monitoring, and tune quantization and KV-cache settings against your own evaluation set.

Nanobase AI is headquartered in Silicon Valley and is a member of the NVIDIA Inception Program. Explore our solutions or book a live demo to see a sized deployment running.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.