Monitoring AI services with Prometheus, Grafana and OpenTelemetry works by instrumenting the application to emit metrics and traces in an open standard format, then scraping, storing and visualizing that data with infrastructure teams already trust. OpenTelemetry provides the instrumentation layer, capturing spans for each model call, retrieval step or tool invocation along with attributes like token counts, latency and model name, and exports them to a backend such as Tempo or Jaeger for trace visualization. Prometheus scrapes time-series metrics, for example requests per second, GPU utilization from NVIDIA DCGM exporters, queue depth on a vLLM or TensorRT-LLM server, and p50 or p99 latency, storing them for alerting through Alertmanager. Grafana then ties these together into dashboards showing GPU memory pressure alongside application-level latency and error rate, which is essential for diagnosing whether a slowdown is caused by GPU saturation, network contention or the model itself. This combination is the natural choice for a team that already runs Prometheus and Grafana for infrastructure monitoring and wants AI-specific signals in the same system rather than a separate observability product. Nanobase AI builds these dashboards directly on top of existing Prometheus and Grafana deployments so AI workloads inherit the same alerting and on-call processes as the rest of the stack.
Which metrics actually predict an AI incident
Teams that bolt Prometheus onto an AI service often scrape whatever the framework exposes by default, then wonder why dashboards look busy but do not explain outages. The signals worth instrumenting split into three layers: hardware, serving engine and application.
| Metric | Source | Why it matters |
|---|---|---|
| GPU utilization and memory | DCGM exporter | Shows saturation vs idle capacity |
| GPU temperature and power draw | DCGM exporter | Flags thermal throttling early |
| Inference queue depth | vLLM or TensorRT-LLM exporter | Backlog forms before users feel latency |
| KV-cache utilization | vLLM metrics endpoint | High eviction caps throughput even with free GPU compute |
| p50 and p99 request latency | OpenTelemetry spans | p99 catches the slow tail averages hide |
| Tokens generated per second | Application metrics | Direct measure of usable capacity |
| Request error and timeout rate | Application metrics | Separates model quality issues from infra issues |
Queue depth and KV-cache utilization catch capacity problems earlier than latency alone, because latency only rises after the backlog has already formed.
Structuring a dashboard that engineers actually open
A dashboard mixing GPU counters, serving-engine internals and business metrics on one screen becomes noise. A more useful layout puts hardware on a top row, serving-engine signals such as queue depth and KV-cache on a second row, and application latency and error rate on a third, so on-call scans top to bottom.
Grafana's variable templating lets one dashboard serve multiple GPU nodes instead of duplicating panels per node. Teams running GPU infrastructure through Kubernetes GPU Operator or Slurm typically already have DCGM exporters deployed cluster-wide, so this dashboard just adds a row.
A dashboard organized by layer, not by data source, is what shortens time to diagnosis during an incident.
An alerting rule that fires before users notice
Alerting on raw latency thresholds alone tends to fire too late. A better pattern pairs a threshold with a sustained duration, so transient spikes do not page anyone while a real degradation still triggers within minutes.
groups:
- name: llm-service-alerts
rules:
- alert: HighP99InferenceLatency
expr: histogram_quantile(0.99, sum(rate(vllm:e2e_request_latency_seconds_bucket[5m])) by (le)) > 5
for: 5m
labels:
severity: warning
annotations:
summary: "p99 inference latency above 5s for 5 minutes"The exact metric name depends on the exporter, but the histogram_quantile pattern over a rate window, gated by for:, applies across vLLM, TensorRT-LLM and custom instrumentation.
A latency alert without a for: duration guard is one of the most common causes of alert fatigue on AI on-call rotations.
Where OpenTelemetry earns its place alongside metrics
Prometheus metrics tell an engineer that p99 latency spiked at 14:02; they do not say which request or step caused it. OpenTelemetry closes that gap by capturing a span per model call, retrieval step or tool invocation, tagged with token count, model name and cache hit status, exported to a trace backend such as Tempo or Jaeger.
During an incident, the workflow starts from the Grafana panel showing the spike, then jumps into the trace backend filtered to that window to find the slow spans, rather than treating metrics and traces as tools owned by different teams. This pairing is what makes it possible to answer the broader question of monitoring an LLM application in production with evidence rather than a guess.
Metrics tell you something changed; traces tell you which request and which step, and incident response is only fast when both are wired together.
Common cardinality mistakes that break this stack
Labeling Prometheus metrics with a per-user ID, per-prompt hash or per-session identifier is the most common way teams overload their time-series database, since Prometheus stores a new series per unique label combination. That detail belongs in traces or logs, not metric labels. A second mistake is scraping at one-second intervals across many GPUs without adjusting retention, inflating storage cost without adding diagnostic value.
High-cardinality labels and overly frequent scraping are the two most common ways a Prometheus deployment for AI workloads becomes unmanageable within a few months.
Frequently asked questions
Do we need OpenTelemetry if we already use Prometheus?
Yes, for anything beyond basic uptime monitoring. Prometheus answers whether something is wrong; OpenTelemetry captures the structured, per-request detail needed to answer why, including which retrieval step or tool call caused a slow or failed response. Most production AI stacks run both rather than choosing one.
How many GPU metrics should we actually alert on?
Start with three: memory utilization approaching capacity, sustained high temperature indicating throttling risk, and queue depth exceeding a threshold that predicts latency degradation. Other GPU metrics such as power draw or clock speed are useful for dashboards and postmortems, but rarely warrant a page-worthy alert on their own.
Can this stack run without Kubernetes?
Yes. Prometheus, Grafana and OpenTelemetry collectors run as standalone services or Docker containers, and the DCGM exporter runs directly on a bare-metal GPU host without any orchestration layer. Kubernetes eases scaling across many nodes, but a single-server or small-cluster deployment does not require it at all.
What is a reasonable Prometheus retention period for AI metrics?
Fifteen to thirty days of full-resolution data is typical for active troubleshooting, with longer-term trends downsampled into a remote-write backend such as Thanos or Mimir. Retaining a full year at one-second resolution for every GPU is rarely worth the storage cost.
How Nanobase AI helps
Nanobase AI, an enterprise AI engineering company headquartered in Silicon Valley, builds Prometheus, Grafana and OpenTelemetry stacks that plug directly into the GPU infrastructure it deploys, so DCGM, vLLM and TensorRT-LLM metrics land in the same dashboards and alerting rules covering the rest of a client's infrastructure. We size the metric set and alert thresholds to the actual serving stack rather than shipping generic templates. See a working example in a demo.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.