How do I autoscale LLM inference on Kubernetes?

Autoscaling LLM inference on Kubernetes usually combines two layers: pod-level autoscaling that adds or removes vLLM or NIM replicas based on load, and node-level autoscaling that provisions or releases GPU nodes to match. For pod scaling, standard CPU-based Horizontal Pod Autoscaler metrics are a poor fit for GPU inference, so most production setups use KEDA with a Prometheus scaler tracking vLLM's own queue depth, GPU utilization, or request latency metrics instead, scaling up before latency degrades rather than reacting to CPU usage that stays flat regardless of GPU load. At the node level, Karpenter or the cluster autoscaler provisions GPU instances on demand, but this needs to account for the real cold-start cost of an inference pod, which includes pulling a multi-gigabyte container image and loading model weights that can take one to several minutes, meaning naive scale-to-zero often produces unacceptable latency spikes for the first requests after scale-up. A common mitigation is keeping a small warm pool of always-on replicas sized for baseline traffic and scaling additional capacity only for peaks, alongside pre-pulling images and caching model weights on fast local storage to cut cold-start time. Nanobase AI, a Silicon Valley Kubernetes infrastructure company, designs these warm-pool and scaling policies around each customer's real traffic variability.

Why CPU-based autoscaling metrics are the wrong signal here

The Horizontal Pod Autoscaler's default CPU utilization metric was designed for stateless web services, and it fails almost completely for GPU inference: a vLLM pod under heavy load can show flat, low CPU usage while its GPU is fully saturated and its request queue is growing, because the actual work is happening on the GPU, not the CPU. Scaling decisions based on CPU utilization in this scenario simply never fire when they should.

This is the first thing to fix before building any autoscaling policy: the signal driving scale decisions has to reflect GPU-side reality, not the CPU metric Kubernetes checks by default.

CPU utilization is close to meaningless as an autoscaling signal for GPU inference workloads, and using it anyway is the most common reason autoscaling setups fail silently.

Pod-level scaling with real signals

KEDA (Kubernetes Event-Driven Autoscaling) replaces the default HPA metric source with custom scalers, most commonly a Prometheus scaler reading metrics vLLM or NIM already expose: request queue depth, GPU utilization, or p95 latency. Queue depth is usually the most actionable signal, since it grows before latency visibly degrades, giving the autoscaler a chance to add capacity ahead of user-facing slowdown rather than reacting after it.

A practical scaling policy pattern:

1. Export vLLM's Prometheus metrics (queue depth, running requests, GPU cache usage) to your monitoring stack.
2. Configure a KEDA ScaledObject targeting queue depth or GPU utilization against your vLLM deployment.
3. Set a minimum replica count above zero for latency-sensitive services, and a cooldown period long enough to avoid thrashing on short traffic spikes.
4. Test the scaling trigger under a synthetic load ramp before trusting it in production.

Queue depth as the primary scaling signal catches load growth before it becomes visible latency degradation, which is the entire point of autoscaling in the first place.

Node-level scaling and the cold-start problem

Pod scaling alone does nothing if there is no GPU node available to schedule the new pod onto, which is where Karpenter or the cluster autoscaler provisions GPU instances on demand. The complication specific to LLM inference is cold-start cost: pulling a multi-gigabyte container image and loading model weights into GPU memory commonly takes one to several minutes, during which any request routed to the new pod either fails or queues, undermining the whole point of scaling up in the first place.

Naive scale-to-zero on GPU inference commonly produces a multi-minute latency spike for the first requests after scale-up, which is why a warm pool sized for baseline traffic is standard practice rather than an optimization.

Sizing the warm pool against real traffic

The warm pool should be sized against your actual baseline load, not zero and not peak capacity, with additional replicas scaling in only for load beyond that baseline. Getting this wrong in either direction has a clear cost: too small a warm pool means frequent cold starts during normal daily traffic variation, too large means paying for idle GPU capacity that never gets used. Reviewing actual traffic patterns over at least a few weeks before setting the warm pool size avoids guessing from assumptions that do not match real usage.

A warm pool sized from actual observed baseline traffic, not a round number picked in advance, is what keeps both cold-start risk and idle GPU cost under control.

Frequently asked questions

Can KEDA scale based on vLLM's own metrics directly?

Yes, vLLm exposes Prometheus-compatible metrics including queue depth and GPU cache usage, which KEDA's Prometheus scaler can consume directly to drive scaling decisions without needing a separate custom metrics pipeline.

How long does GPU node provisioning typically take?

This varies by cloud provider and instance type availability, often ranging from under a minute to several minutes depending on the GPU type and region capacity; this delay is separate from and additive to the model cold-start time on the new node.

Should we ever scale GPU inference pods to zero?

Only for genuinely intermittent workloads where occasional multi-minute latency on the first request after idle is acceptable, such as internal batch tools used a few times a day. User-facing production services almost always warrant a nonzero minimum replica count.

Does the NVIDIA GPU Operator affect autoscaling?

The GPU Operator manages driver and device plugin lifecycle on nodes, which autoscaling depends on being correctly installed on any newly provisioned GPU node; it does not itself drive scaling decisions but needs to be configured to bootstrap automatically on new nodes.

How Nanobase AI helps

Nanobase AI, a Silicon Valley Kubernetes infrastructure company, designs warm-pool sizing and KEDA scaling policies around each customer's real traffic variability rather than default templates, and configures the GPU Operator to bootstrap new nodes automatically as part of the same autoscaling pipeline. See our solutions for the full Kubernetes GPU infrastructure practice.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.