How do I monitor GPU utilization with DCGM and Prometheus?

You monitor GPU utilization with DCGM and Prometheus by running NVIDIA's dcgm-exporter, a container that reads metrics from the Data Center GPU Manager daemon and exposes them in Prometheus format on an HTTP endpoint that Prometheus scrapes on a regular interval. On Kubernetes, the GPU Operator can deploy dcgm-exporter automatically as a DaemonSet so every GPU node reports metrics without manual setup, and you point your existing Prometheus configuration at it the same way you would any other exporter. The default metric set covers GPU and memory utilization, temperature, power draw, clock speed, PCIe throughput, Xid error counts, and NVLink bandwidth, and you can customize which fields are collected through a metrics configuration file if the defaults are too broad or too narrow for your dashboards. Once metrics land in Prometheus, Grafana's official DCGM dashboard gives a working starting point showing per-GPU utilization and temperature trends across the fleet, which you then extend with cluster-specific panels such as per-team GPU-hours. This combination is the de facto standard stack for GPU observability, since DCGM is NVIDIA's own supported tool rather than a community reimplementation of GPU telemetry. Nanobase AI deploys DCGM, Prometheus, and Grafana as the default observability stack on every GPU cluster it operates.

Key DCGM metrics worth tracking

Utilization alone tells you whether a GPU is busy, not whether it is doing useful work efficiently, so pairing it with memory, power, and Xid metrics gives a far more actionable picture than utilization in isolation.

Deploying dcgm-exporter and scraping it

On Kubernetes, the GPU Operator can deploy dcgm-exporter automatically as a DaemonSet, so every GPU node reports metrics without a manual install step. Point an existing Prometheus configuration at it the same way you would any other exporter, using a standard scrape job.

scrape_configs:
  - job_name: 'dcgm-exporter'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: nvidia-dcgm-exporter
        action: keep

The default metric set covers utilization, memory, temperature, power, clock speed, PCIe throughput, Xid counts, and NVLink bandwidth, and a metrics configuration file lets you narrow or expand that set if the defaults are too broad or too narrow for a specific dashboard.

Building useful Grafana panels

Grafana's official DCGM dashboard is a solid starting point, showing per-GPU utilization and temperature trends across the fleet, but most teams extend it with cluster-specific panels once the base dashboard is running. Useful additions include per-team or per-namespace GPU-hours consumed, a heatmap of utilization across the entire fleet to spot idle nodes at a glance, and a dedicated Xid error panel that surfaces new error codes immediately rather than burying them in a general log stream.

Cardinality and retention at scale

A fleet running hundreds of GPUs across dozens of nodes generates meaningfully more time series than a typical application monitoring setup, since DCGM exports many metrics per individual GPU rather than one aggregate per node. Left unmanaged, this can strain Prometheus storage and query performance faster than teams expect coming from application-level monitoring. Reducing scrape frequency for less time-sensitive metrics, using the metrics configuration file to trim fields you do not actually query, and setting appropriate retention windows for raw high-cardinality data are all worth doing before the metric volume becomes a problem rather than after.

Frequently asked questions

Does dcgm-exporter work the same way outside Kubernetes?

Yes, dcgm-exporter can run as a standalone container or service on any host with the DCGM daemon running, and Prometheus scrapes it the same way regardless of whether Kubernetes or a traditional Slurm-managed node is underneath. The only requirement is that the DCGM daemon itself be installed and reachable on that host, since dcgm-exporter is simply a thin metrics layer on top of it.

How often should Prometheus scrape GPU metrics?

A common starting interval is every 10 to 30 seconds for most metrics, tightened for latency-sensitive production inference monitoring and loosened for less time-sensitive fields to manage overall time series volume at scale. The right interval ultimately depends on how quickly you need to detect a problem versus how much cardinality your Prometheus storage can absorb.

Can I monitor MIG instances individually with DCGM?

Yes, DCGM exposes per-MIG-instance metrics distinctly from whole-GPU metrics once MIG is enabled, letting you track utilization and memory for each partitioned instance separately rather than only at the physical GPU level. This granularity matters in multi-tenant clusters, where knowing which specific MIG slice is under-utilized or overloaded is far more actionable than an aggregate figure for the whole card.

Is the default Grafana DCGM dashboard enough for production?

It is a strong starting point but usually needs extension with cluster-specific context like per-team attribution and dedicated alerting panels, since the default dashboard focuses on raw per-GPU telemetry rather than organizational usage patterns. Teams running production clusters typically layer on cost-attribution views and Xid-specific panels once the base dashboard has proven the underlying data pipeline works.

How Nanobase AI helps

Nanobase AI deploys DCGM, Prometheus, and Grafana as the default observability stack on every GPU cluster it operates, tuning metric scope and retention to each customer's fleet size before cardinality becomes a problem. For deciding what to actually alert on from these metrics, see what GPU metrics to alert on in production.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.