In production, you should alert on GPU utilization dropping unexpectedly during an active job, GPU temperature exceeding safe thresholds typically above 85 to 90 degrees Celsius depending on the model, ECC memory errors and Xid error codes, power throttling events, and NVLink or network link errors that indicate a fabric problem rather than a compute problem. Sudden low utilization on a GPU that should be busy is one of the most actionable alerts, since it often means a data loader bottleneck, a stalled distributed training rank, or a job that silently fell back to CPU execution. Xid errors reported through DCGM are worth alerting on individually rather than aggregating, since specific codes map to specific hardware conditions, with certain codes typically indicating a GPU that has fallen off the bus and needs a physical reseat or replacement. Double-bit ECC errors should trigger immediate alerts because they indicate memory corruption risk rather than a transient issue, while single-bit errors are usually just logged and tracked for trend. Power and thermal alerts protect hardware longevity and catch cooling failures before they cause throttling that silently slows every job on that node. Nanobase AI configures these alert thresholds as part of its GPU cluster monitoring setup, tuned to each customer's hardware generation and workload pattern.
Alert tiers by severity
| Metric | Trigger guidance | Severity | Typical action |
|---|---|---|---|
| Sudden low utilization on a busy job | Drops well below expected baseline unexpectedly | High | Investigate data loader, stalled rank, or CPU fallback |
| GPU temperature | Above roughly 85-90°C, model-dependent | High | Check cooling, consider power capping |
| Double-bit ECC error | Any occurrence | Critical | Cordon node, schedule hardware inspection |
| Xid error codes | Specific codes indicating hardware fault | Critical to high, code-dependent | Cordon and route to diagnostics per code |
| Power throttling event | Any sustained occurrence | Medium | Check cooling and power delivery |
| NVLink or network link errors | Rising error counters on a link | High | Investigate cable, transceiver, or switch port |
Sudden low utilization on a GPU that should be busy is one of the most actionable alerts available, since it almost always points to a specific, fixable cause rather than ambiguous background noise.
Why Xid errors deserve individual alerting
Aggregating Xid errors into a single generic counter hides the signal that makes them useful, since specific Xid codes map to specific hardware conditions rather than one undifferentiated "something is wrong" state. A code typically indicating a GPU has fallen off the PCIe bus needs an entirely different response, a physical reseat or hardware replacement, than a code indicating a recoverable driver-level fault that a restart resolves. Alerting on individual codes, or at minimum grouping them by known severity category, lets an on-call engineer respond correctly on the first pass instead of investigating from scratch every time.
What not to page on
Alert fatigue is a real cost, and paging on every metric fluctuation trains engineers to ignore alerts entirely. Single-bit ECC errors are typically logged and trended rather than paged on immediately, since they are expected at some background rate and only become concerning as a rising trend rather than an isolated event. Brief, momentary utilization dips during normal job phase transitions, such as checkpoint writes or data loader batch boundaries, are also usually not worth an immediate page, and tuning alert thresholds to tolerate this normal variance is what keeps the alerting system trusted rather than muted.
Tying alerts to automated remediation
- Detect the triggering condition through DCGM or the exporter feeding your alerting pipeline.
- Classify the severity, distinguishing a software-recoverable event from one requiring physical hardware intervention.
- For hardware-indicating events, automatically cordon the affected node in your scheduler so no new work lands there.
- Drain any currently running job safely rather than killing it abruptly, where the scheduler supports a graceful drain.
- Route the node into a repair or diagnostics workflow, and only return it to the pool after DCGM diagnostics confirm it is healthy again.
Automating the cordon step specifically, rather than relying on a human to react to a page before a failing GPU corrupts or slows an active job, is the single highest-leverage piece of this pipeline to build first.
Frequently asked questions
What temperature threshold should trigger an alert?
Roughly 85 to 90 degrees Celsius is a common starting point for data center GPUs, but the exact safe threshold varies by model and cooling design, so consult your specific GPU's thermal specifications rather than applying one number universally. It also helps to pair a fixed threshold with a rate-of-change check, since a GPU climbing rapidly toward that limit often signals a cooling problem worth investigating before the threshold is even reached.
Should every Xid code trigger an immediate page?
No. Some Xid codes indicate benign, recoverable events while others indicate serious hardware faults; grouping codes by known severity and paging only on the serious category avoids overwhelming on-call engineers with low-value alerts. Lower-severity codes are still worth logging and trending, since a rising frequency of an otherwise benign code can itself become a signal worth escalating.
How do I avoid alert fatigue on a large GPU fleet?
Tune thresholds to tolerate normal operational variance, group related low-severity events into digest notifications rather than individual pages, and reserve immediate paging for conditions that genuinely require prompt human action. Periodically reviewing which alerts actually led to action, and retiring or adjusting ones that never do, keeps the system calibrated as the fleet grows.
Can GPU alerting be integrated with existing incident management tools?
Yes, Prometheus Alertmanager and similar tools route alerts from DCGM metrics into standard incident management and paging systems the same way any other infrastructure alert would be routed. This means GPU-specific alerts can inherit the same on-call rotations, escalation policies, and runbook links your team already uses for other infrastructure, rather than requiring a separate parallel alerting workflow just for GPU hardware.
How Nanobase AI helps
Nanobase AI, a Silicon Valley enterprise AI engineering company, configures these alert thresholds as part of its GPU cluster monitoring setup, tuned to each customer's hardware generation and workload pattern rather than applying generic defaults. For the underlying metrics pipeline, see monitoring GPU utilization with DCGM and Prometheus.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.