What is FP8 and FP4 support on Hopper and Blackwell GPUs?

FP8 and FP4 are reduced precision numeric formats that let GPUs run AI workloads faster and with less memory by representing weights and activations with fewer bits than the traditional FP16 or FP32 formats. FP8 was introduced with the Hopper architecture on the H100 and H200 through NVIDIA's first generation Transformer Engine, using two variants called E4M3 and E5M2 that trade off precision and dynamic range, and it roughly halves memory footprint compared to FP16 while maintaining accuracy close to higher precision formats when properly calibrated. FP4 arrived with the Blackwell architecture on the B200 and B300 through a second generation Transformer Engine, cutting memory and bandwidth requirements even further and enabling notably higher throughput, though it demands more careful quantization and calibration since dropping to four bits per value increases the risk of accuracy degradation on sensitive layers if done naively. Both formats require serving stack support, and frameworks like TensorRT-LLM and vLLM have progressively added FP8 and FP4 kernels to exploit this hardware capability. Choosing FP8 versus FP4 in production involves a real accuracy versus throughput tradeoff that should be validated against the specific model and task. Nanobase AI, a Silicon Valley enterprise AI engineering company, evaluates FP8 and FP4 quantization for clients against actual accuracy requirements rather than defaulting to the fastest option.

Fewer bits per number, more numbers per second

FP8 and FP4 are reduced-precision floating point formats that trade numeric precision for speed and memory efficiency, representing weights and activations with 8 or 4 bits instead of the 16 bits used by FP16 or the 32 bits used by FP32. Halving the bit width roughly halves the memory footprint for the same tensor, which directly addresses the memory-bandwidth-bound nature of LLM inference, while also letting Tensor Cores process more values per clock cycle, increasing raw throughput.

The two formats were introduced a generation apart and are not interchangeable defaults; each requires calibration and serving-stack support to use safely in production.

FP8 versus FP4: generation, format, and risk profile

Why FP8 became a practical default on Hopper

FP8's two common variants, E4M3 and E5M2, split the available bits differently between the exponent and mantissa, letting engineers pick a variant that favors either dynamic range or precision depending on where a tensor sits in the network. Because NVIDIA's first-generation Transformer Engine automates much of the scaling and calibration work, teams running Hopper hardware were able to adopt FP8 inference broadly without extensive manual tuning, and it now maintains accuracy close to higher-precision formats for most transformer workloads when calibrated properly.

FP4 pushes further in the same direction but with a narrower margin for error. Dropping to four bits per value increases the risk of accuracy degradation on sensitive layers, such as those handling outlier activations or attention score computation, if quantization is applied naively across the whole model. NVIDIA's second-generation Transformer Engine on Blackwell is built specifically to manage this, tracking numeric ranges more granularly and applying FP4 selectively where it is safe, but the technique demands more careful validation before production use than FP8 typically does.

What actually determines whether FP8 or FP4 is usable

1. Serving framework support: TensorRT-LLM, vLLM, and similar engines have progressively added FP8 and FP4 kernels, but exact support varies by model architecture and framework version, so this should be verified for the specific stack in use.
2. Model sensitivity: some architectures and layers tolerate aggressive quantization better than others, which means accuracy validation against the actual task, not just a general benchmark, is necessary before committing to FP4 in production.
3. Hardware generation: FP8 runs on both Hopper and Blackwell; FP4 requires Blackwell-generation hardware and its second-generation Transformer Engine to get the intended throughput and accuracy management.
4. Calibration effort: FP4 generally demands more careful calibration data and validation than FP8, which has a longer track record of working well with lighter tuning effort.

Frequently asked questions

Can FP8 and FP4 run on the same GPU?

Blackwell GPUs like the B200 support both FP8 and FP4, giving flexibility to choose per workload, while Hopper GPUs like the H100 and H200 support FP8 but not the newer FP4 format introduced with Blackwell's second-generation Transformer Engine.

Does FP4 always outperform FP8 for inference throughput?

FP4 generally offers higher raw throughput and lower memory footprint than FP8 on the same Blackwell hardware, but the accuracy tradeoff is also larger, so the right choice depends on validating the specific model's tolerance for aggressive quantization against the task's accuracy requirements.

Do we need to manually implement FP8 or FP4 quantization?

Not entirely; the NVIDIA Transformer Engine automates much of the scaling and calibration work for both formats, and serving frameworks like TensorRT-LLM and vLLM increasingly expose FP8 and FP4 as configuration options rather than requiring custom kernel work.

Is FP4 safe for every model type?

No, sensitivity varies by architecture and by which layers handle outlier values. Validating accuracy on representative evaluation data before deploying FP4 in production is important rather than assuming it is a drop-in replacement for FP8.

How Nanobase AI helps

Nanobase AI evaluates FP8 and FP4 quantization against a client's actual accuracy requirements and serving stack rather than defaulting to the fastest available option. We validate precision choices on representative workloads as part of deployment, drawing on our work comparing vLLM, TensorRT-LLM, Ollama, and SGLang. Explore our GPU infrastructure and inference solutions or contact us to review your model's precision tradeoffs.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.