How do I benchmark NCCL bandwidth between GPUs?

You benchmark NCCL bandwidth between GPUs using nccl-tests, the official NVIDIA test suite that includes binaries such as all_reduce_perf and all_gather_perf built directly against your installed NCCL, MPI, and CUDA versions. Build the suite with MPI support enabled if you plan to test across multiple nodes, then run the all-reduce benchmark across a range of message sizes, since bandwidth ramps up differently at small versus large payload sizes and small-message latency matters more for some workloads than peak bandwidth. On a single node with NVLink, expect bus bandwidth close to the link's rated speed, roughly 900 GB/s aggregate on an H100 NVLink domain, while multi-node results depend heavily on whether InfiniBand or RoCE is correctly detected, confirmed by checking the debug output for the chosen transport. Compare results against vendor reference numbers for your exact GPU and network generation, since a result 20 percent or more below reference usually points to a cabling, firmware, or PCIe topology issue rather than a benchmark artifact. Running this test during cluster acceptance, before customer workloads start, catches problems early. Nanobase AI, headquartered in Silicon Valley, runs nccl-tests as a standard step in every cluster commissioning process.

Building and running the benchmark

nccl-tests is NVIDIA's official suite and must be built against the exact CUDA, NCCL, and MPI versions installed on the cluster, since a mismatched build produces misleading or failing results that have nothing to do with actual hardware performance.

1. Clone nccl-tests and build with MPI support enabled if testing across multiple nodes, since single-node builds skip the inter-node code path entirely.
2. Run all_reduce_perf across a range of message sizes from a few kilobytes to several gigabytes, since bandwidth ramps up differently at small versus large payloads.
3. For multi-node runs, launch through MPI or your scheduler's native multi-node launch mechanism with one process per GPU.
4. Capture the debug log alongside the benchmark output to confirm which transport, NVLink, InfiniBand, or RoCE, was actually used for the run.
5. Repeat across different node pairs and node counts to catch inconsistent links rather than trusting a single run.

mpirun -np 16 -hostfile hosts.txt ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

Building against the wrong CUDA or NCCL version is the most common reason a benchmark run produces numbers nobody can explain, so always confirm the build matches what production workloads will actually use.

What to expect at different scales

These expected ranges give the baseline against which any specific cluster's measured results should be judged.

Interpreting results below reference

A result 20 percent or more below the vendor reference figure for your exact GPU and network generation is a signal to investigate, not a benchmark artifact to ignore. Start by checking that GPUDirect RDMA is actually active and that the network adapter shares a PCIe switch with the GPU rather than crossing the CPU's own interconnect, since poor PCIe placement is a common and easily overlooked cause of underperformance even on fully compatible hardware. Next, check firmware versions on host channel adapters and switches, since a firmware mismatch across an otherwise identical fleet can silently cap throughput on specific nodes.

When to run this test

Running nccl-tests during cluster acceptance, before any customer workload starts, catches cabling, firmware, and topology problems while they are still cheap to fix, rather than after a training job has already been running for days and throughput looks suspiciously low. Re-running the same benchmark periodically, and after any hardware change such as a cable swap or firmware update, catches regressions before they show up as an unexplained slowdown in a production training run.

Frequently asked questions

Do I need MPI to run nccl-tests across multiple nodes?

Yes, or an equivalent multi-process launch mechanism; nccl-tests relies on MPI to coordinate ranks across nodes for the multi-node benchmark, though single-node tests can run without it using local process spawning. Slurm and other schedulers can substitute for a manual MPI launch as long as they set the same per-node process and rank environment variables.

What message size should I focus on for LLM training workloads?

Large message sizes, typically hundreds of megabytes to several gigabytes, are most representative of gradient synchronization traffic in large model training, though checking the full range still helps catch latency issues affecting smaller collective operations. Focusing the benchmark there mirrors the payload sizes an all-reduce actually moves during backpropagation.

Can I use nccl-tests results to predict actual training throughput?

It gives a strong signal about network health but not a direct throughput prediction, since actual training also depends on compute time per step, overlap between computation and communication, and the specific parallelism strategy in use. Treat a clean nccl-tests result as a necessary baseline rather than a guarantee of end-to-end training speed.

How often should nccl-tests be re-run on a production cluster?

At minimum after any hardware change, firmware update, or driver upgrade, and periodically as a standing health check, since a fabric that passed acceptance testing can still degrade over time from a failing cable or a switch port developing errors.

How Nanobase AI helps

Nanobase AI, headquartered in Silicon Valley, runs nccl-tests as a standard step in every cluster commissioning process, comparing results against vendor reference figures for the exact GPU and network generation before any customer workload is onboarded. For troubleshooting errors that show up afterward, see what is NCCL and how to troubleshoot errors.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.