Why does memory bandwidth matter more than TFLOPS for LLM inference?

Memory bandwidth matters more than TFLOPS for LLM inference because autoregressive token generation is fundamentally memory bound rather than compute bound: to produce each new token, the GPU must read the entire set of model weights, and often the growing KV cache, from memory, while performing comparatively little arithmetic per byte moved. This gives large language model decoding very low arithmetic intensity, meaning the GPU's compute units sit idle waiting on memory transfers even though its theoretical TFLOPS rating looks impressive on paper. That is why a GPU like the H200, with 141 GB of HBM3e at about 4.8 TB/s, can outperform a GPU with higher peak FLOPS but lower bandwidth on real inference workloads, and why batching multiple requests together helps, since it amortizes the same weight read across more useful computation. Prefill, the initial processing of the input prompt, is more compute bound and benefits more from raw TFLOPS, so overall throughput depends on the mix of prompt length and generation length. Engines like vLLM and TensorRT-LLM are built specifically to exploit this by maximizing batch size and memory reuse. Nanobase AI, an NVIDIA Inception Program member, sizes GPU infrastructure around memory bandwidth and capacity first, treating TFLOPS as a secondary consideration for inference heavy deployments.

The arithmetic intensity problem

Autoregressive decoding does very little math per byte of data moved: to produce one token, the GPU reads the entire weight set (and growing KV cache) from memory but performs comparatively little arithmetic on it, which is the textbook definition of a memory-bandwidth-bound workload rather than a compute-bound one. This is why a GPU's headline TFLOPS number, which measures peak arithmetic throughput, correlates poorly with real decode speed, while HBM bandwidth correlates closely.

Bandwidth versus compute across current GPUs

Two GPUs with similar peak FLOPS but different bandwidth will produce noticeably different tokens-per-second on decode-heavy workloads, which is why bandwidth, not FLOPS, is the number to check first when sizing inference hardware.

Prefill versus decode changes the picture

Not all of inference behaves the same way. Prefill, the initial pass that processes the input prompt, performs much more arithmetic per byte of memory accessed because it processes many tokens in parallel through the same weights, making it comparatively more compute bound. Decode, generating each subsequent token one at a time, is where the memory-bound behavior dominates. A workload's overall throughput profile depends on the ratio of prompt length to generation length: long-prompt, short-answer workloads lean more compute bound during prefill, while long-generation, short-prompt workloads (like chat or reasoning traces) lean heavily memory bound.

Why batching is the standard countermeasure

Batching multiple requests together is the primary way serving engines fight the memory-bound nature of decode: since every batch still has to read the same weights from memory once, processing several requests' worth of tokens against that single read amortizes the fixed memory cost across more useful computation. This is the core mechanism behind continuous batching in engines like vLLM and TensorRT-LLM, and it is why GPUs with more memory (to hold bigger batches and larger KV caches) often out-throughput GPUs with higher FLOPS but tighter memory.

What this means for hardware selection

1. Check HBM or GDDR bandwidth before TFLOPS when comparing inference GPUs for chat-style or reasoning workloads with long generations.
2. Check total memory capacity next, since it caps how large a batch (and therefore how much amortization) is possible before hitting an out-of-memory error.
3. Treat TFLOPS as more relevant for prefill-heavy workloads (long prompts, short answers) or training, where arithmetic intensity is higher.
4. Validate with an actual serving benchmark, since scheduler quality and KV cache management in the serving engine affect how well theoretical bandwidth translates into real throughput.

For a deeper look at how this plays out between specific GPUs, see H100 vs H200 specifications and our guide on vLLM vs TensorRT-LLM vs Ollama vs SGLang.

Frequently asked questions

Does higher TFLOPS ever matter more than bandwidth for inference?

Yes, for prefill-heavy workloads with long prompts and short outputs, or for tasks like embedding generation and classification that process many tokens in parallel without long autoregressive generation, compute throughput matters more relative to bandwidth.

Why does batching help GPUs with more memory more than GPUs with less?

More memory allows larger batch sizes and bigger KV caches before hitting capacity limits, and since the fixed cost of reading weights from memory is amortized across the whole batch, GPUs that can hold bigger batches get more useful computation per memory read.

Is GDDR memory ever competitive with HBM for inference?

GDDR-based cards like the RTX PRO 6000 or L40S offer meaningfully lower bandwidth than HBM-based data center GPUs, which limits achievable throughput under high concurrency, but they remain viable for smaller models or lower-concurrency deployments where bandwidth headroom is less critical.

Does quantization change the memory bandwidth equation?

Yes. Running weights in FP8 or INT4 rather than FP16 roughly halves or quarters the bytes that must be read per token, which directly reduces the memory-bandwidth bottleneck and is one reason lower-precision serving delivers such large real-world throughput gains.

How Nanobase AI helps

Nanobase AI, an accepted member of the NVIDIA Inception Program, sizes GPU infrastructure around memory bandwidth and capacity first, treating TFLOPS as a secondary consideration for inference-heavy deployments, and validates that sizing against real serving benchmarks rather than spec sheets. See our approach to inference infrastructure.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.