How do I deploy vLLM on Kubernetes?

Deploying vLLM on Kubernetes typically means packaging it as a container with the NVIDIA GPU Operator installed on the cluster, requesting GPU resources through a Deployment or the vLLM production-stack Helm chart, and exposing the OpenAI-compatible server through a Service and an Ingress or gateway. The vLLM project publishes official Docker images and a Kubernetes-native production stack that adds a router for multi-replica load balancing, KV cache aware routing, and Prometheus metrics for autoscaling decisions, which saves significant integration work compared with writing manifests from scratch. Key configuration choices include setting resource requests and limits to match GPU memory, mounting a persistent volume or object storage for model weights so pods start quickly, setting readiness probes that account for model load time of one to several minutes, and choosing tensor-parallel size based on how many GPUs each pod should span. For multi-node tensor or pipeline parallelism, LeaderWorkerSet or a similar StatefulSet pattern coordinates the ranks. Health checks, graceful shutdown handling for in-flight requests, and node affinity that keeps GPU pools isolated from general workloads round out a production setup. Nanobase AI, an NVIDIA Inception program member, builds these Kubernetes GPU Operator and vLLM deployments end to end for enterprise clusters.

The configuration choices that break a first deployment

Most first attempts at running vLLM on Kubernetes fail not because of GPU scheduling, which the NVIDIA GPU Operator handles reliably, but because of assumptions carried over from stateless web service deployments. Model loading takes anywhere from tens of seconds to several minutes depending on model size and storage throughput, which breaks default readiness probe timing. In-flight requests need graceful handling during rollouts, which breaks default termination behavior. And GPU memory is a hard allocation, not a soft resource, which breaks assumptions from CPU-based autoscaling.

Get these three specifics right before anything else, since they cause the majority of production incidents in early vLLM-on-Kubernetes deployments.

Configuration checklist for the pod spec

A minimal deployment shape

resources:
  limits:
    nvidia.com/gpu: 2   # matches --tensor-parallel-size 2
readinessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 90
  periodSeconds: 10
terminationGracePeriodSeconds: 120

This is illustrative, not complete: a real deployment adds volume mounts for model weights, environment variables for engine arguments, and resource requests matched to the node's GPU memory.

Why the official production stack saves real time

Writing these manifests from scratch works for a single-replica deployment, but production traffic usually needs multiple replicas behind a router that understands vLLM's internals, specifically one that can route requests to the replica most likely to have relevant KV cache already warm, rather than a naive round robin. The vLLM project's own production-stack Helm chart provides this router along with Prometheus metrics wiring, which is meaningfully different from a generic load balancer that treats every replica as interchangeable.

Steps for a first production rollout

1. Install the NVIDIA GPU Operator on the cluster and confirm nvidia.com/gpu resources are visible and schedulable.
2. Build or pull an official vLLM container image pinned to a tested version.
3. Deploy using the production-stack Helm chart rather than hand-written manifests for anything beyond a single replica.
4. Configure persistent storage or a shared cache for model weights so pod restarts do not re-download multi-gigabyte files.
5. Set readiness probes and termination grace periods based on measured model load time and maximum request duration, not framework defaults.
6. Wire Prometheus metrics into existing cluster monitoring and set alerts on queue depth and GPU memory utilization rather than only CPU and pod restarts.
7. Test a rolling update under simulated load before relying on it during real traffic, confirming in-flight requests complete cleanly.

Multi-node deployments need a different pattern

For models requiring multi-node tensor or pipeline parallelism, a standard Deployment does not coordinate rank assignment across pods correctly. A StatefulSet-based pattern, such as LeaderWorkerSet, is needed so each node knows its rank and the cluster can address them as a coordinated group rather than independent replicas. This matters specifically for very large models that exceed a single node's GPU count, covered in more depth in running vLLM with multiple GPUs and in Kubernetes GPU Operator vs Slurm for the broader orchestration choice.

Frequently asked questions

Do we need the NVIDIA GPU Operator to run vLLM on Kubernetes?

Yes, in almost all cases, since it installs and manages the device plugin, drivers, and monitoring components that expose GPUs as schedulable Kubernetes resources; without it, pods cannot reliably request and receive GPU allocations.

How should autoscaling work for a vLLM deployment?

Scale on request queue depth or GPU utilization rather than CPU usage, since CPU is rarely the bottleneck for GPU-bound inference; a custom metrics adapter reading vLLM's own Prometheus metrics is the common pattern for this.

What happens to in-flight requests during a rolling update?

If terminationGracePeriodSeconds is set too short, in-flight generations get killed mid-stream when a pod is replaced; setting it to exceed your maximum expected request duration lets the old pod finish serving before termination.

Can multiple vLLM replicas share the same model weights on disk?

Yes, mounting a shared read-only volume or object storage cache for model weights across replicas avoids redundant downloads and speeds up scaling events, which matters more as replica count and model size grow.

How Nanobase AI helps

Nanobase AI builds and operates production vLLM deployments on Kubernetes, from GPU Operator installation and node pool design through Helm-based rollout, health check tuning, and autoscaling wired to real inference metrics. This is core to Nanobase AI's NVIDIA GPU infrastructure practice as an NVIDIA Inception Program member.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.