H100 vs H200 vs B200: which GPU for LLM inference, and when?

For most enterprise LLM inference in 2026, the H100 is still the right buy for models up to 70B served in FP8; the H200 is the better choice when a 400B-class model or DeepSeek R1 must fit on one 8-GPU node, or when long-context KV cache dominates memory; the B200 or GB200 NVL72 is justified only for the highest throughput per rack, FP4 serving, or a 72-GPU NVLink domain for very large MoE models. H100 and H200 share the same compute, so the H200 gain is purely memory and bandwidth. Size for memory first, bandwidth second and TFLOPS last.

H100 vs H200 vs B200: spec comparison

H100 and H200 are both Hopper with identical compute; the H200 swaps 80 GB of HBM3 for 141 GB of HBM3e. B200 is Blackwell: FP4 support, roughly double the bandwidth again, double the NVLink. Figures are NVIDIA published specifications for SXM and HGX parts as of 2026; check the official H100, H200 and HGX pages, since specs are revised periodically.

Key takeaway: H100 and H200 share compute, so you pay for memory and bandwidth; B200 changes compute, precision and interconnect too.

Why memory and bandwidth decide inference speed, not TFLOPS

Prefill (reading the prompt) is compute-bound. Decode (emitting tokens one at a time) is bound by how fast the GPU streams weights and KV cache out of HBM, and chat and agent workloads spend most of their time in decode. Weights set the floor: a 70B dense model needs about 140 GB in FP16, 70 GB in FP8 and 38 GB in INT4; a 400B-class model about 810, 405 and 220 GB; DeepSeek R1 and V3 (671B MoE, released in FP8) about 700 GB.

KV cache is what teams underestimate. For Llama 3 70B each token costs about 320 KB in FP16, so one 128K-token sequence needs about 40 GB; FP8 KV cache halves that. Plan 20 to 50 percent headroom above weights for short-context chat and compute KV cache explicitly above 32K. As a spec ratio rather than a benchmark, memory-bound decode has about 1.4x the bandwidth headroom on H200 versus H100 and about 2.4x on B200.

Key takeaway: size for weights plus KV cache first; within a generation, bandwidth ratio is the best first-order predictor of decode throughput.

Decision matrix by model size and workload

Assumptions: FP8 weights, FP8 or FP16 KV cache, vLLM or TensorRT-LLM, and 25 to 40 percent of memory reserved for KV cache under concurrency. Chat means short prompts at high concurrency; long-context means 64K to 128K+ tokens per request; batch means offline throughput.

The 70B row is where the H200 first pays for itself: one H200 replaces two H100s and removes tensor-parallel traffic. The 671B row is where the H100 drops out: DeepSeek R1 in FP8 does not fit in 640 GB. Per-model counts are worked through in How many GPUs do you need for 70B, 405B and DeepSeek R1?. The 70B-on-one-H200 configuration in vLLM, leaving roughly 55 GB for KV cache:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

Key takeaway: up to 70B, H100 is sufficient; 70B to 405B is H200 territory; 671B MoE at high concurrency is where Blackwell earns its premium.

When the H100 is still the right buy

As of 2026 the H100 is the most mature part in the lineup: every serving stack and quantization kernel has been tuned on it for three years, refurbished systems exist, and cloud capacity is abundant. If your models are 70B or smaller and context stays under about 32K, an H100 fleet serves them in FP8 with headroom. It is the right buy when most of these hold:

• Models up to 70B in FP8, or up to 35B in FP16, at short to medium context.
• Multi-tenant serving where MIG partitions (up to 7 per GPU) matter more than one very large model.
• Air-cooled racks at 10 to 12 kW and no budget for facility work.
• Fine-tuning limited to LoRA or QLoRA.
• Price sensitivity: as of 2026 H100 systems are the cheapest of the three per server, H200 carries a moderate premium and B200 the largest with longer lead times; verify current pricing and run the numbers in Your own GPUs instead of cloud APIs.

An RTX PRO 6000 Blackwell (96 GB GDDR7, PCIe, no NVLink) is a cheaper single card for models up to 35B at modest concurrency, but with roughly half the bandwidth of an H100 it is not a substitute under load.

Key takeaway: buy H100 when your largest model fits comfortably on two GPUs in FP8 and your room is air-cooled; it is the lowest-risk, best-supported option.

When the H200's extra memory pays off

The H200 is the same GH100 silicon with 141 GB of HBM3e at 4.8 TB/s: 76 percent more memory and 43 percent more bandwidth at the same 700 W. In HGX form it is a drop-in for H100, so an H100 platform usually accepts H200 modules with a firmware update. Three situations make the memory decisive.

1. One large model per node. A 400B-class model in FP8 needs about 405 GB; on 8x H100 that leaves under 235 GB for KV cache and runtime, on 8x H200 about 720 GB.
2. DeepSeek R1 and V3. In native FP8 they fit on one 8x H200 node and not on one 8x H100 node, which removes inter-node expert-parallel traffic and halves the servers you operate.
3. Long context at concurrency. At about 40 GB of KV cache per 128K sequence on a 70B model, an H100 pair saturates after a handful of concurrent long requests; an H200 pair serves roughly twice as many at the same latency.

The H200 NVL (PCIe, 141 GB, up to 600 W, 4-way NVLink bridge) is the simplest way to put a 70B FP8 model on one card in an existing chassis.

Key takeaway: choose H200 when weights plus KV cache do not fit on H100 without adding GPUs, or when you serve 405B or DeepSeek-class models on one node.

When B200 or GB200 NVL72 is justified

Blackwell changes three things at once: FP4 with a second-generation Transformer Engine, about 8 TB/s of HBM3e bandwidth and 1.8 TB/s of NVLink per GPU. The B200 ships in HGX B200 and DGX B200 systems (8 GPUs, x86 hosts, air- or liquid-cooled). The GB200 NVL72 puts 72 Blackwell GPUs and 36 Grace CPUs in one liquid-cooled rack with a single NVLink domain of about 13.5 TB of HBM3e, so a 671B MoE runs across dozens of GPUs without InfiniBand. Blackwell is justified when one of these binds:

• Throughput per rack or per watt is the limit: about 2.4x the decode bandwidth of H100 per GPU, and FP4 halves weight traffic again where the model tolerates it.
• You serve 400B+ dense or 671B MoE models at high concurrency and long context, where the 72-GPU NVLink domain removes the inter-node bottleneck.
• You run disaggregated prefill and decode at scale, or full-parameter fine-tuning of 70B+ models.

Two cautions. NVIDIA's published speedups for GB200 NVL72 versus H100 assume FP4, NVLink-domain-scale parallelism and specific configurations; plan on a smaller, workload-dependent gain and benchmark your own model. vLLM and TensorRT-LLM support Blackwell, FP4 weights and FP8 KV cache as of 2026, but the newest kernels are the least tested (engine trade-offs: vLLM vs TensorRT-LLM vs Ollama vs SGLang). Blackwell Ultra (B300, GB300, 288 GB per GPU) is also shipping as of 2026; the logic is unchanged, the memory ceiling moves up.

Key takeaway: B200 is justified by throughput per rack and FP4, GB200 NVL72 by the single 72-GPU NVLink domain; if neither binds, H200 is the better value.

PCIe vs SXM, power and cooling

PCIe vs SXM

SXM modules sit on an HGX baseboard with NVSwitch, giving every GPU 900 GB/s (Hopper) or 1.8 TB/s (Blackwell) to every other GPU; that is what makes tensor parallelism across 4 or 8 GPUs efficient and is mandatory for 400B-class models and training. SXM parts run at the full 700 W or about 1,000 W and need a purpose-built server. PCIe cards (H100 PCIe 80 GB at 350 W, H100 NVL 94 GB, H200 NVL 141 GB at up to 600 W) fit standard servers and draw less power, but without a bridge GPUs talk over PCIe Gen5 at about 64 GB/s per direction, roughly one-fourteenth of Hopper NVLink, so tensor parallelism beyond a bridged pair or quad stalls on communication.

Key takeaway: SXM for tensor parallelism of 4 or more, training and 400B+; PCIe for single-card and paired serving, lower power, MIG multi-tenancy and existing servers.

Power and cooling

An 8-GPU HGX H100 or H200 server draws about 10 to 11 kW at full load (DGX H100 and H200 are rated at 10.2 kW), above the 8 to 10 kW many enterprise racks were built for, so one server per rack or a rear-door heat exchanger is common. A DGX B200 is rated at about 14.3 kW and can be air-cooled, but two per rack needs containment or liquid assist. GB200 NVL72 is liquid-cooled by design at about 120 kW per rack and needs facility water, coolant distribution units and a floor rated for the weight. Confirm rack power and cooling before choosing the GPU tier, plan 20 percent headroom above nameplate, treat anything above about 40 kW per rack as a liquid-cooling project, and size 400 Gb/s InfiniBand per GPU for multi-node clusters; the full stack is in the on-premise LLM deployment guide.

Key takeaway: H100 and H200 fit air-cooled rooms with modest upgrades; B200 is the limit of air; GB200 NVL72 is a facilities decision as much as a GPU decision.

Sizing checklist before you commit

1. List the models you will serve in the next 12 months: parameter count, dense or MoE, target precision.
2. Compute weight memory (parameters times bytes per weight) and add 20 to 50 percent for KV cache and runtime, or compute KV cache explicitly above 32K context.
3. Set latency and concurrency targets (time to first token, tokens per second per user, concurrent sessions) and decide whether decode or prefill dominates.
4. Pick the smallest tier that fits on one node with headroom: H100 up to 70B, H200 up to 405B and DeepSeek R1, B200 or GB200 beyond that or for FP4 throughput.
5. Decide SXM versus PCIe from the tensor-parallel degree you need, not from price.
6. Verify rack power, cooling and floor loading against the server nameplate with 20 percent headroom.
7. Benchmark the actual model on the actual engine before purchase; a short cloud rental of the same GPU type is cheap next to a mis-sized order.

Key takeaway: size from measured weights, KV cache and latency targets, then pick the lowest tier that fits on one node.

Frequently asked questions

Is the H200 faster than the H100 for LLM inference?

Only in memory-bound work. The H200 has the same compute as the H100 but 141 GB of HBM3e at 4.8 TB/s versus 80 GB at 3.35 TB/s. Decode streams weights and KV cache every token, so high-concurrency chat and long context gain from the 43 percent higher bandwidth; prefill-heavy summarisation sees little change. Benchmark your own model rather than trusting headline figures.

Can DeepSeek R1 run on 8x H100?

Not in native FP8 on one node. DeepSeek R1 and V3 are 671B MoE models that need about 700 GB for FP8 weights, more than the 640 GB in an 8x H100 server. The options are two H100 nodes over InfiniBand, INT4 quantization with a quality evaluation, or one 8x H200 (1,128 GB) or 8x B200 node, which is what most teams choose as of 2026.

Do I need a B200 to serve a 70B model?

No. A 70B model in FP8 needs about 70 GB of weights and serves well on two H100s or one H200. A B200 is faster per GPU thanks to about 2.4x the memory bandwidth of H100 and FP4, but that only pays when you need maximum tokens per second per rack or are consolidating many 70B replicas. For one production deployment, H100 or H200 is the economical choice.

What is the difference between B200 and GB200?

The B200 is a Blackwell GPU sold in 8-GPU HGX B200 and DGX B200 systems with x86 hosts. The GB200 pairs one Grace CPU with two Blackwell GPUs over NVLink-C2C; 36 of them form a GB200 NVL72 rack with 72 GPUs in one NVLink domain, liquid-cooled at about 120 kW. Choose B200 for conventional servers and GB200 NVL72 when one model must span dozens of GPUs without InfiniBand.

Is a PCIe H100 good enough for inference?

Yes, for models that fit on one card or a bridged pair. The H100 PCIe (80 GB HBM2e, 2 TB/s, 350 W) and H100 NVL (94 GB HBM3, 3.9 TB/s) serve 8B to 35B models in FP16 and 70B in FP8 across two bridged cards. Beyond a pair, tensor parallelism over PCIe Gen5 is the bottleneck, so 400B-class models and training belong on SXM.

How much power and cooling does an 8-GPU server need?

About 10 to 11 kW for an air-cooled 8x H100 or H200 server (DGX H100 and H200 are rated at 10.2 kW), about 14 to 15 kW for a DGX B200, and about 120 kW per rack for a liquid-cooled GB200 NVL72. Add 20 percent headroom for power distribution, confirm the room can reject the heat, and plan liquid cooling above about 40 kW per rack.

Does FP8 or FP4 quantization reduce answer quality?

FP8 weights and FP8 KV cache on Hopper and Blackwell are close to lossless for most open-weight models, and DeepSeek R1 and V3 are trained in FP8 natively. FP4 (NVFP4 on Blackwell) and INT4 halve memory again but need a quality evaluation on your own tasks, especially reasoning and code. Use FP8 by default, FP4 or INT4 after measurement, FP16 only when memory is not a constraint.

How Nanobase AI can help

Nanobase AI sizes, installs and operates NVIDIA GPU infrastructure for private LLM deployment: model-to-GPU sizing across H100, H200, B200 and RTX PRO systems, PCIe versus SXM selection, rack power and cooling, NVLink and InfiniBand topology, Kubernetes GPU Operator or Slurm scheduling, MIG partitioning and monitoring. On top we deploy vLLM, TensorRT-LLM or NVIDIA NIM with FP8 and FP4 quantization, RAG and fine-tuning, validated with benchmarks on your own models before you place an order. Headquartered in Silicon Valley and an accepted member of the NVIDIA Inception Program, we deliver on-premise, in AWS, Azure and Google Cloud, or hybrid. See our solutions or book a live demo.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.