You run health checks and burn-in on new GPU nodes by combining NVIDIA's DCGM diagnostics tool, which runs a structured series of increasingly intensive tests from a quick check up to a full multi-hour stress test, with sustained synthetic workloads that exercise compute, memory bandwidth, and interconnect simultaneously to surface marginal hardware before it fails in production. Start with the deepest DCGM diagnostic level, which stresses GPU memory, runs targeted CUDA kernels, and checks NVLink and PCIe bandwidth against expected reference values, flagging GPUs that pass a quick check but fail under sustained load. Follow that with a multi-hour or overnight burn-in using a stress tool or a representative training workload at full node count, since some failures, particularly thermal throttling or marginal power delivery, only appear after sustained high utilization rather than in a short test. For multi-node clusters, also run nccl-tests across the full fabric during burn-in to catch bad cables or switch ports that a single-node test would miss entirely. Log every result against vendor reference specifications so a node that passes at only 90 percent of expected NVLink bandwidth, for example, gets flagged rather than silently accepted. Nanobase AI, headquartered in Silicon Valley, runs this full health check and burn-in sequence on every node before handing a cluster over to a customer.
Acceptance testing needs a written bar, not a gut check
A common failure pattern with new GPU hardware is accepting a node because it "passed" without a specific, written threshold defining what passing means. A GPU that clears a five-minute quick check can still fail under sustained load, so a real acceptance process needs escalating test depth and numeric thresholds tied to vendor reference specifications, not a pass or fail impression.
DCGM diagnostic levels
| Level | Duration | What it tests | When to run |
|---|---|---|---|
| r1 (quick) | Under a minute | Basic GPU presence, driver health | Fast sanity check after any reboot |
| r2 | A few minutes | Memory bandwidth, basic compute kernels | Post-provisioning first pass |
| r3 | Around 10-15 minutes | Targeted CUDA kernels, PCIe bandwidth | Pre-production validation |
| r4 (full) | Multiple hours | Sustained stress, NVLink/PCIe bandwidth under load, thermal behavior | Final acceptance before production |
Running only the quick level and declaring a node healthy is the single most common gap in informal acceptance processes; marginal power delivery and thermal throttling issues characteristically surface only under the sustained load that r4 or an overnight burn-in applies.
A burn-in runbook
- Run DCGM level r1 immediately after OS and driver installation to catch gross hardware or driver failures before investing further test time.
- Run level r3 once the node is fully provisioned, checking PCIe and NVLink bandwidth against the vendor's published reference figures for that specific GPU and server SKU.
- Run the full r4 diagnostic, or a multi-hour synthetic stress workload at full node utilization, to surface thermal throttling and marginal power delivery that shorter tests miss.
- For multi-node clusters, run
nccl-testsacross the complete network fabric at full node count during burn-in, not just single-node tests, since a bad cable or degraded switch port only shows up when traffic actually crosses that link. - Run an overnight or multi-hour representative training workload at full utilization as a final real-world check, since some failures only appear after sustained production-like load rather than synthetic stress alone.
- Log every measured result against vendor reference specifications in a structured record, flagging any node achieving less than expected bandwidth (for example, meaningfully below expected NVLink throughput) rather than silently accepting it.
Acceptance criteria worth writing down explicitly
A written acceptance standard should specify minimum NVLink and PCIe bandwidth as a percentage of vendor reference values, maximum acceptable GPU temperature under sustained r4 load, zero uncorrected ECC errors during the full test cycle, and successful completion of nccl-tests at expected bandwidth across every node pair in the target topology, not just adjacent nodes. Without these numbers in writing, "the node passed diagnostics" becomes a subjective judgment call that different engineers on the team will apply inconsistently.
Frequently asked questions
How long should burn-in take for a full rack of new nodes?
A thorough burn-in sequence, including full DCGM diagnostics and an overnight sustained workload, typically adds one to three days per rack depending on node count and how quickly failures are triaged, which should be built into deployment timelines rather than compressed to meet a launch date.
What happens if a node fails burn-in?
A failed node should be flagged for vendor RMA or reseating and cable reinspection depending on the specific failure, then re-run through the full sequence after remediation rather than assumed fixed. Nodes that fail repeatedly on the same test point to a hardware defect rather than a transient issue.
Is burn-in necessary for cloud GPU instances too?
Cloud providers generally handle hardware validation themselves, but running a quick DCGM check and an nccl-tests bandwidth check on newly provisioned multi-node cloud GPU instances is still worthwhile, since cloud hardware occasionally ships with degraded components that a quick validation catches before a training job wastes hours on a bad node.
How Nanobase AI helps
Nanobase AI, headquartered in Silicon Valley, runs this full DCGM and nccl-tests burn-in sequence with documented acceptance criteria on every node before handing a cluster to a customer, catching marginal hardware before it becomes a production incident weeks into training.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.