Low GPU utilization during training almost always traces back to the GPU sitting idle while waiting on something else, most commonly a data loading pipeline that cannot read, decode, and preprocess samples as fast as the GPU can consume them, followed by inefficient checkpointing, small batch sizes, or communication overhead in distributed training. Start by checking whether utilization dips are periodic and aligned with data loading, which points to too few data loader worker processes, slow storage, or CPU-bound preprocessing such as image decoding or tokenization that should be moved to a faster format or precomputed offline. In distributed multi-GPU training, low utilization can also mean GPUs are waiting on a slow all-reduce, which happens when the network fabric is underperforming, when one node is a straggler due to thermal throttling, or when gradient synchronization is not overlapped with backward-pass computation. Small batch sizes relative to GPU memory leave compute capacity unused, so increasing batch size or using gradient accumulation can help if memory allows. Profiling tools built into PyTorch or Nsight Systems show exactly where time goes in each training step, distinguishing compute time from data-loading and communication stalls. Nanobase AI profiles and tunes training pipelines for customers whose GPU utilization falls well below the 80 to 90 percent that well-tuned jobs typically achieve.
Read the shape of the graph before touching code
Most teams open nvidia-smi or a Grafana panel, see a number below 50 percent, and start guessing. The shape of the utilization curve over time tells you which category of problem you have before you profile anything. A sawtooth pattern that drops to zero at regular intervals almost always means the GPU is starved between steps, usually by data loading or checkpoint writes. A flat line sitting steadily at, say, 40 percent without dropping to zero points to a batch size or kernel efficiency problem rather than starvation. A single node or GPU consistently lower than its peers in a multi-node job points to a straggler, not a global issue.
| Pattern | Likely cause | First thing to check |
|---|---|---|
| Sawtooth, drops to 0% each step | Data loader can't keep up | iostat, dataloader worker count |
| Sawtooth aligned with save intervals | Synchronous checkpoint writes | Checkpoint write duration vs step time |
| Flat, steady 30-60% | Batch too small, kernel inefficiency | Increase batch size, check torch.compile |
| One GPU/node lagging the rest | Straggler: thermal, PCIe, or NCCL | dcgmi diag, nccl-tests on that node |
| Drops correlate with collective ops | Network fabric contention | NCCL debug logs, switch counters |
Instrument before you guess
Reaching for nsys profile or PyTorch's built-in profiler on a short representative run (typically 20 to 50 steps is enough) gives a timeline that separates compute kernels, host-to-device copies, and communication collectives. This matters because a graph that looks like "low utilization" from nvidia-smi can actually be a GPU that is 100 percent busy on a badly fused kernel doing unnecessary work, which no amount of data loader tuning will fix. NCCL_DEBUG=INFO on a distributed job adds visibility into collective timing and will surface a ring or tree topology that is not matching your actual interconnect.
For data-loading suspicions specifically, py-spy dump against a running training process shows whether worker processes are stuck in a decode or tokenization call, which is the most common single cause of sawtooth utilization on vision and long-context text pipelines alike.
Common root causes ranked by frequency
- Too few data loader workers or CPU-bound preprocessing — increase
num_workers, move image decode or tokenization to a faster library, or precompute and cache preprocessed samples offline. - Storage throughput below what workers can request — check whether the dataset sits on local NVMe versus a network filesystem under load from other jobs.
- Small per-GPU batch size — raise batch size or use gradient accumulation if memory allows; small batches leave streaming multiprocessors underfed regardless of how fast the rest of the pipeline runs.
- Unoverlapped gradient synchronization — verify the training framework overlaps all-reduce with backward-pass computation rather than blocking on it, which is standard in modern DDP and FSDP but can regress with custom training loops.
- A single slow or thermal-throttled node — DCGM diagnostics and a quick
nccl-testsbandwidth run isolate whether one node is dragging down an entire distributed job.
When the fix is architectural, not tactical
Sometimes low utilization is not a bug but a structural mismatch between the workload and the cluster's network topology. Tensor-parallel training run across nodes without a low-latency, non-blocking fabric will show low utilization no matter how well the data pipeline is tuned, because every layer's forward pass blocks on a cross-node all-reduce. In that case, the fix is either reducing cross-node tensor parallelism in favor of more pipeline or data parallelism, or upgrading the interconnect, not further profiling of the training script.
Frequently asked questions
What GPU utilization percentage should a well-tuned training job hit?
Well-tuned single-node training jobs typically sustain 80 to 90 percent GPU utilization. Multi-node jobs with heavy cross-node communication can run somewhat lower even when correctly configured, so compare against a known-good baseline for your specific model and topology rather than a universal target.
Does high utilization always mean the job is running efficiently?
No. A GPU can show 100 percent utilization while running an inefficient, unfused kernel or waiting on many small memory-bound operations. Utilization measures whether the GPU is busy, not whether the work it is doing is optimal, which is why profiling tools that break down kernel time matter more than the utilization number alone.
Can mixed precision training affect utilization readings?
Yes indirectly. Moving to FP16 or BF16 reduces compute time per step, which can make data loading or communication stalls, previously hidden behind longer compute time, suddenly visible as utilization drops, even though nothing about the pipeline got slower in absolute terms.
How Nanobase AI helps
Nanobase AI profiles training pipelines end to end, from data loader through NCCL collectives, to find exactly where GPU-hours are being wasted rather than guessing at a fix. That includes tuning data pipelines, checkpoint scheduling, and network topology together, since the three interact more often than teams expect. Explore how we approach this on our solutions page or see it applied to your own workload in a live demo.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.