What is the difference between FP16, BF16, FP8, and INT4?

FP16, BF16, FP8 and INT4 differ in how many bits they use to represent each number and therefore in memory footprint, numeric range and precision. FP16 uses 16 bits with a 5-bit exponent and 10-bit mantissa, giving good precision but a limited numeric range that can cause overflow during training; BF16 also uses 16 bits but with an 8-bit exponent matching FP32's range, trading some precision for better numerical stability, which is why it is the default for training and widely used for inference too. FP8 uses 8 bits, roughly halving memory versus FP16 or BF16, and is natively accelerated on Hopper and Blackwell tensor cores in formats like E4M3 and E5M2, making it a strong default for inference on H100, H200 and B200 GPUs. INT4 uses 4 bits through post-training quantization methods like AWQ or GPTQ, cutting memory to about a quarter of FP16, with a larger but often acceptable accuracy trade-off depending on the task. Choosing between them is a direct trade-off between GPU memory, inference speed and output quality. Nanobase AI, a Silicon Valley enterprise AI engineering company, selects the right precision format for a customer's hardware and accuracy requirements rather than defaulting to one option.

Bit layout and numeric range compared

Each format allocates its bits differently between sign, exponent (numeric range) and mantissa (precision within that range), and that allocation is what drives every practical difference in memory, speed and accuracy.

Key takeaway: BF16 trades mantissa precision for FP32-level range, which is exactly why it largely replaced FP16 for both training and much of inference.

Hardware support by GPU generation

Not every format runs equally fast on every GPU, and native tensor-core acceleration is what turns a smaller format into a real speed advantage rather than just a memory saving.

An A100 can technically run FP8-quantized weights, but without hardware acceleration for the format the speed benefit is far smaller than on Hopper or Blackwell, where FP8 tensor cores deliver close to double the throughput of FP16 at half the memory.

Key takeaway: FP8's speed advantage is a Hopper-and-newer feature, so the same quantized model behaves differently on an A100 than it does on an H100.

Matching format to task

Training generally uses BF16 or mixed precision with FP32 master weights, since gradient accumulation benefits from BF16's wide range and training is less sensitive to the memory savings that smaller formats offer. Inference is where the format choice really matters for cost: FP16 remains useful when reproducing a research baseline or when a checkpoint has no validated lower-precision version, FP8 is the practical default for production inference on Hopper and Blackwell given its native acceleration and small accuracy cost, and INT4 is reserved for cases where memory or hardware footprint is the binding constraint and some accuracy trade-off is acceptable after validation.

Key takeaway: BF16 for training, FP8 as the default for production inference on modern hardware, INT4 when memory footprint is the overriding constraint.

What the format choice means in real GPU count

Translating these formats into a concrete example makes the trade-off tangible: a 70B model needs about 140 GB in FP16, 70 GB in FP8 and 38 GB in INT4. On 80 GB H100 hardware, that is the difference between needing two GPUs at FP16, one GPU with modest headroom at FP8, or one GPU with generous headroom at INT4. The format decision is rarely made in the abstract; it is made against a specific GPU budget, and stating the target hardware alongside the precision choice is what turns this comparison from a reference table into an actual purchasing decision.

Key takeaway: precision choice and GPU count are the same decision viewed from two sides, so pick the format against your actual target hardware, not in isolation. The accuracy side of the FP8 choice specifically is covered in does FP8 reduce model quality.

Frequently asked questions

Why does BF16 have fewer mantissa bits than FP16 if it's considered better for training?

BF16 sacrifices mantissa precision (7 bits versus FP16's 10) specifically to gain exponent range (8 bits versus FP16's 5), which prevents the overflow and underflow issues that FP16 can suffer during training gradient computation. The precision trade-off matters less than the stability gain in most training workloads.

Is FP8 one format or several?

FP8 has two common variants: E4M3 (4 exponent bits, 3 mantissa bits), favoring precision over range and commonly used for weights and activations, and E5M2 (5 exponent bits, 2 mantissa bits), favoring range and sometimes used for gradients during training.

Can I mix precisions within the same model?

Yes, this is common practice: for example, FP8 weights with an FP8 or FP16 KV cache, or a model where sensitive layers are kept at higher precision while most layers run in a lower one. Serving engines and quantization libraries increasingly support this kind of mixed configuration.

Does INT4 use the same exponent-mantissa structure as the floating-point formats?

No, INT4 represents values as integers within a fixed range determined by a scale factor computed during quantization, rather than using a floating-point exponent and mantissa. This is a fundamentally different representation, which is part of why INT4 accuracy behavior differs from simply "FP8 but smaller."

How Nanobase AI helps

Nanobase AI, a Silicon Valley enterprise AI engineering company, selects the right precision format for a customer's specific hardware generation and accuracy requirements, rather than defaulting to whichever format a model ships with by default. We validate the chosen format's throughput and accuracy on your workload before it reaches production. Our H100 vs H200 vs B200 guide covers which hardware generation supports which format natively.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.