You detect GPU failures and Xid errors by monitoring kernel logs and DCGM output for Xid codes, which are NVIDIA driver error codes logged whenever a GPU encounters a hardware or driver-level fault, ranging from relatively benign transient events to a GPU falling off the PCIe bus entirely. DCGM's health check module and dcgm-exporter both surface Xid counts as a metric you can alert on directly, and cross-referencing the specific code against NVIDIA's published reference table tells you whether the event indicates a software issue, likely resolved by a driver restart, or a hardware issue requiring node maintenance. Common actionable patterns include codes that typically signal a GPU falling off the bus, needing a physical reseat, cable check, or RMA, and other codes that often point to uncorrectable ECC memory errors. Once a failure is confirmed, the standard response is to cordon the affected node in your scheduler, drain running jobs safely, and route the node into a repair workflow using DCGM diagnostics or NVIDIA's field diagnostic utilities. Automating this cordon-on-Xid response prevents a failing GPU from silently corrupting or slowing an active job. Nanobase AI, a Silicon Valley enterprise AI engineering company, builds automated Xid detection and node quarantine workflows into the clusters it manages.
What Xid codes are and where they show up
Xid codes are NVIDIA driver error codes logged whenever a GPU encounters a hardware or driver-level fault, ranging from relatively benign transient events to a GPU falling off the PCIe bus entirely. They appear in kernel logs directly and are surfaced as a metric through DCGM's health check module and dcgm-exporter, which is the more practical place to monitor them at fleet scale rather than tailing kernel logs node by node.
Common Xid codes worth knowing
The table below covers a few widely referenced examples; consult NVIDIA's published Xid reference for the complete, authoritative list before making a hardware decision based on a code alone.
| Xid pattern | General category | Typical response |
|---|---|---|
| GPU has fallen off the bus | Hardware or PCIe connection fault | Physical reseat, cable check, or RMA |
| Double-bit ECC error | Uncorrectable memory error | Cordon node, schedule memory diagnostics |
| Graphics engine exception | Driver or application-level fault | Often resolved by driver or process restart |
| GPU memory page fault | Application memory access issue | Investigate application code before assuming hardware fault |
Treating every Xid code as equally serious is a mistake in both directions: it either causes unnecessary node replacements for recoverable software faults or, worse, causes teams to under-react to genuinely hardware-indicating codes buried in the same alert stream.
An automated detection pipeline
- Configure DCGM's health check module and dcgm-exporter to surface Xid counts as a first-class Prometheus metric rather than relying on log scraping alone.
- Cross-reference each occurring code against NVIDIA's published reference table to classify it as software-recoverable or hardware-indicating.
- For hardware-indicating codes, trigger an automatic cordon on the affected node so the scheduler stops placing new work there.
- Drain any currently running job on that node gracefully, giving it a chance to checkpoint before the node exits service.
- Route the cordoned node into a repair workflow, running DCGM diagnostics or NVIDIA's field diagnostic utilities to confirm the specific fault before deciding between a reseat, cable replacement, or RMA.
Automating this pipeline end to end, rather than relying on someone to notice a stray kernel log line, is what turns Xid detection into a fleet-wide safeguard instead of a per-incident scramble.
Deciding reseat, restart, or RMA
A code typically associated with a GPU falling off the bus usually points to a physical connection issue, a loose PCIe seating, a failing cable, or in some cases a genuinely failed card, and the first response is a physical reseat and cable check before escalating to an RMA request. A code associated with a driver-level exception rather than a bus disconnection often resolves with a driver or node restart, without needing physical hardware intervention at all. The practical discipline here is trying the cheap fix first, restart or reseat, and only escalating to a full RMA process once diagnostics confirm the fault persists across that cheaper remediation.
Preventing recurrence with burn-in testing
Running DCGM diagnostics and a sustained burn-in test on every new GPU node before it enters production catches marginal hardware, cards that pass a quick check but fail under sustained load, before it becomes a mid-training-run failure. This matters especially for newly delivered hardware, where a small percentage of units commonly show early-life failures that burn-in testing catches in hours rather than after a costly multi-day training run has already been interrupted.
Frequently asked questions
Does every Xid error mean the GPU needs to be replaced?
No. Many Xid codes indicate software-recoverable conditions resolved by a driver or process restart, while a smaller set of codes genuinely indicate hardware faults needing physical intervention. Classifying the specific code correctly is essential before deciding on a response. Consulting NVIDIA's Xid reference for the exact code observed is the fastest way to tell which category a given event falls into.
Where is the authoritative list of Xid codes and their meanings?
NVIDIA publishes and maintains an Xid error reference as part of its driver documentation; always consult the current version for your driver release rather than relying on a fixed table, since code meanings and new codes are documented there directly.
Can Xid errors be caught before they affect a running job?
Continuous DCGM monitoring with alerting catches many Xid events close to when they occur, but some hardware faults, particularly a sudden bus disconnection, will still interrupt whatever job was actively using that GPU at the moment of failure. For workloads that checkpoint frequently, this window matters less, since the job can resume from a recent checkpoint on a healthy node rather than losing significant progress.
How does burn-in testing reduce Xid-related failures in production?
It surfaces marginal hardware, units that function under light load but fail under sustained stress, before that hardware ever runs a real workload, catching early-life failures during a controlled test window instead of during a production training run. Running this check consistently across every incoming node also builds a track record of which batches or suppliers show elevated early-failure rates over time.
How Nanobase AI helps
Nanobase AI, an accepted member of the NVIDIA Inception Program, builds automated Xid detection and node quarantine workflows into the clusters it manages, pairing DCGM-based classification with a defined reseat-restart-RMA decision path. For the broader monitoring stack this depends on, see what GPU metrics to alert on in production.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.