NVLink is NVIDIA's proprietary high speed interconnect that lets GPUs communicate directly with each other at far higher bandwidth and lower latency than standard PCIe, which matters enormously for multi GPU servers running large models. Fourth generation NVLink on the H100 delivers about 900 GB/s of bidirectional bandwidth per GPU, while fifth generation NVLink on the B200 roughly doubles that, compared to a PCIe Gen5 x16 link that tops out around 128 GB/s bidirectional. This bandwidth gap matters because training and serving large language models often requires splitting a model across multiple GPUs, through tensor or pipeline parallelism, which forces constant exchange of activations and gradients between GPUs; without NVLink, that communication becomes the bottleneck and GPUs sit idle waiting for data rather than computing. In systems like HGX and DGX, all eight GPUs on a baseboard are connected through an NVSwitch fabric, effectively giving every GPU full bandwidth access to every other GPU rather than a simple point to point link. This is why GPU counts alone do not predict performance for large model workloads, since the interconnect topology matters as much as the number of GPUs. Nanobase AI, a Silicon Valley enterprise AI engineering company, designs multi GPU clusters around NVLink topology to avoid communication bottlenecks that spec sheets alone would not reveal.
NVLink versus PCIe by the numbers
| Interconnect | Bidirectional bandwidth per GPU | Used in |
|---|---|---|
| PCIe Gen5 x16 | ~128 GB/s | Standard rack servers, PCIe GPUs |
| NVLink 4th gen | ~900 GB/s | H100, H200 (SXM) |
| NVLink 5th gen | ~1.8 TB/s | B200 |
The bandwidth gap is the entire reason NVLink exists: at roughly 7x to 14x the throughput of PCIe Gen5, NVLink turns GPU-to-GPU communication from a bottleneck into a non-issue for workloads that must split a model across multiple GPUs.
Why this matters more than GPU count
Training and serving large language models frequently requires splitting a model across GPUs through tensor or pipeline parallelism, which forces constant exchange of activations and gradients between GPUs during every forward and backward pass. Without a fast interconnect, GPUs spend a large share of their time idle, waiting for data to arrive over a comparatively slow link, rather than computing. This is why two servers with the same GPU count and generation can perform very differently on large-model workloads if one uses NVLink and the other relies on PCIe alone for cross-GPU communication — GPU count alone does not predict performance for sharded workloads.
How the fabric scales across a full server
In HGX and DGX systems, all eight GPUs on a baseboard connect through an NVSwitch fabric rather than simple point-to-point links, which means every GPU gets full NVLink bandwidth to every other GPU in the system, not just to its nearest neighbor. This all-to-all topology is what allows an 8-GPU server to behave like one coherent accelerator for tensor-parallel workloads, and it is the same principle scaled up further in systems like the GB200 NVL72, which extends the NVLink domain across 72 GPUs in a single rack.
When NVLink matters less
Not every workload needs it. Data-parallel training, where each GPU works on a different batch of data mostly independently and only synchronizes occasionally, tolerates PCIe-only connectivity much better than tensor-parallel workloads do. Similarly, serving a model that fits entirely on a single GPU gets no benefit from NVLink at all, since there is no cross-GPU communication to accelerate. This is part of why GPUs like the RTX PRO 6000, which rely on PCIe rather than a full NVLink mesh, remain competitive for single-GPU or lightly parallel inference despite lacking NVLink.
Checklist for evaluating interconnect needs
- Determine whether the target model must be sharded across more than one GPU at the precision you plan to run.
- If yes, identify whether the parallelism style is tensor/pipeline (communication-heavy, needs NVLink) or data-parallel (communication-light, PCIe often sufficient).
- Check whether the candidate server's baseboard provides full NVSwitch connectivity across all GPUs or only pairwise bridging.
- Factor future growth: workloads that fit on one GPU today may not once model size or context length requirements grow.
Frequently asked questions
Does every 8-GPU server include full NVLink between all GPUs?
Only SXM-based systems built on NVIDIA's HGX or DGX baseboard include full NVSwitch connectivity across all GPUs. PCIe-based 8-GPU builds typically only bridge NVLink between adjacent pairs, if at all.
Is NVLink bandwidth the same in both directions?
The bandwidth figures cited (900 GB/s for 4th gen, ~1.8 TB/s for 5th gen) are bidirectional aggregate figures per GPU, representing simultaneous send and receive capacity across the link.
Does NVLink generation matter if I'm only using two GPUs?
It still matters for workloads with heavy cross-GPU traffic between those two GPUs, though the relative benefit is smaller than in an 8-GPU tensor-parallel configuration where communication patterns are more complex.
Can NVLink connect GPUs across different physical servers?
Standard NVLink operates within a single server's baseboard. Extending NVLink-class bandwidth across multiple physical nodes requires purpose-built rack-scale systems like GB200 NVL72; otherwise, inter-server communication relies on InfiniBand or Ethernet.
How Nanobase AI helps
Nanobase AI, an enterprise AI engineering company with engineering headquarters in Silicon Valley, designs multi-GPU clusters around NVLink topology to avoid communication bottlenecks that spec sheets alone would not reveal, matching interconnect requirements to actual parallelism strategy. Learn more about our GPU infrastructure design work.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.