How many tokens per second can an H100 generate?

Tokens per second on an H100 varies enormously with model size, quantization, batch size and whether the measurement is single-user latency or aggregate throughput across many concurrent requests, so there is no single figure that applies broadly. For a small model in the 7B to 8B range at FP8 with continuous batching, aggregate throughput across many concurrent users can reach into the thousands of tokens per second, while single-user generation speed typically lands around 50 to 100 or more tokens per second. A 70B model, even in FP8, generates individual responses more slowly, often in the range of 20 to 40 tokens per second per user, with aggregate throughput across concurrent requests scaling with how much KV cache headroom is available for batching. Larger batch sizes generally raise aggregate throughput at the cost of higher per-request latency, so the right operating point depends on whether the workload prioritizes speed for a single user or total capacity across many users. NVIDIA NIM and vLLM benchmarks for comparable model and hardware combinations are a reasonable starting reference before committing to specific hardware. Nanobase AI, a Silicon Valley enterprise AI engineering company, benchmarks the exact model and quantization on candidate GPUs before making a throughput claim to a customer.

Throughput ranges by model size

Single-user generation speed and aggregate throughput across many concurrent users are different numbers, and both matter depending on what the deployment optimizes for. The ranges below assume FP8 precision with continuous batching.

Key takeaway: single-user speed drops as model size grows, while aggregate throughput depends more on how much KV cache headroom is available to batch requests together.

The factors that move these numbers more than model size alone

Quantization is the first lever: FP8 versus FP16 roughly doubles achievable throughput on Hopper hardware thanks to native FP8 tensor-core acceleration, independent of model size. Batch size is the second: larger batches improve aggregate throughput by keeping the GPU's compute units busier per token, up to the point where compute saturates and further batching only adds latency without more real throughput. Prompt and output length is the third: prefill, processing the input prompt, is compute-bound and scales with prompt length, while decode, generating each output token, is largely memory-bandwidth-bound, so workloads dominated by long prompts and short outputs behave very differently from the reverse. The serving engine itself is the fourth: continuous batching in vLLM or TensorRT-LLM, and the tuned kernels in NVIDIA NIM, can produce meaningfully different throughput on identical hardware and identical model.

Key takeaway: quantization, batch size, prompt/output length ratio and serving engine choice each move throughput independently of model size, which is why headline numbers vary so widely between sources.

Reading a throughput benchmark correctly

1. Check whether the reported number is single-user latency or aggregate throughput across concurrent requests; these answer different questions and are not comparable to each other.
2. Check the precision used, since an FP16 number and an FP8 number for the same model are not directly comparable.
3. Check the prompt and output length assumptions, since a benchmark using short prompts and short outputs will report different numbers than one using longer sequences.
4. Check whether the benchmark used continuous batching or a simpler serving approach, since this materially affects aggregate throughput figures.
5. Treat any number without these details stated as a rough directional estimate rather than a figure to size hardware against directly.

Key takeaway: a throughput number is only meaningful alongside its precision, batching and prompt-length assumptions, which is why benchmarking your own configuration is more reliable than comparing headline figures.

How H200 and B200 shift these figures

H200 shares H100's compute exactly, so single-user tokens-per-second for a model that fits on either GPU changes little; the gain shows up almost entirely in aggregate throughput once concurrency and KV cache headroom become the limiting factor, since H200's 4.8 TB/s bandwidth against H100's 3.35 TB/s lets more requests be batched together before memory runs out. B200 changes the picture on both axes: its higher FP8 compute throughput and roughly 8 TB/s bandwidth lift single-user speed somewhat and aggregate throughput substantially, particularly once FP4 formats like NVFP4 are in use, since Blackwell's dedicated FP4 tensor cores are not available on Hopper at any batch size.

Key takeaway: H200 mainly raises aggregate throughput through extra memory bandwidth, while B200 raises both single-user speed and aggregate throughput through higher compute and native FP4 support.

Frequently asked questions

Why is single-user speed lower for larger models even with the same GPU?

Larger models require more floating-point operations per token, since compute scales with parameter count, so even with identical memory bandwidth and precision, a 70B model takes longer to produce each token than a 7B model does.

Does adding more H100s increase single-user speed or just capacity?

For a model that already fits on one GPU, adding more GPUs primarily increases aggregate capacity for more concurrent users rather than speeding up any individual user's response, unless tensor parallelism is specifically used to split compute for lower per-token latency on a single request.

Is H200 meaningfully faster than H100 for the same model and precision?

H200 shares H100's compute, so single-user generation speed is similar for models that fit on either GPU. The H200's advantage shows up in aggregate throughput at high concurrency and long context, where its larger memory and higher bandwidth allow more requests to be batched together.

What tokens-per-second figure should I use for capacity planning?

Use aggregate throughput measured at your target concurrency and realistic prompt/output lengths, not single-user speed, since capacity planning is about how many total tokens the deployment can produce across all active users, not how fast any one user's response streams.

How Nanobase AI helps

Nanobase AI, a Silicon Valley enterprise AI engineering company, benchmarks the exact model, precision and batching configuration on candidate GPUs before making a throughput claim to a customer, distinguishing clearly between single-user latency and aggregate capacity. Our guide to estimating throughput before buying GPUs covers the benchmarking process in more depth, and our live demo shows this sizing work in action.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.