How do we run vLLM on AWS EKS or Azure AKS with GPUs?

Running vLLM on AWS EKS or Azure AKS with GPUs involves provisioning a GPU enabled node group, typically using P5 or G5 instances on EKS or ND or NC series virtual machines on AKS, installing the NVIDIA device plugin or the fuller NVIDIA GPU Operator so Kubernetes can schedule pods against GPU resources, then deploying vLLM as a containerized service that requests those GPU resources through standard Kubernetes resource limits. The vLLM container needs the model weights available either baked into the image, mounted from persistent storage such as EBS or Azure Files, or pulled from object storage at startup, and its OpenAI-compatible API server should sit behind a Kubernetes service and ingress or load balancer for application traffic. For multi-GPU models, vLLM's tensor parallelism setting needs to match the number of GPUs allocated to each pod, and node affinity rules should keep those GPUs on the same physical instance to avoid cross-node communication overhead. Horizontal pod autoscaling based on request queue depth or GPU utilization, rather than simple CPU metrics, gives more accurate scaling behavior for LLM serving. Monitoring GPU memory and utilization through NVIDIA DCGM exporters integrated with Prometheus is essential for catching out-of-memory issues before they cause outages. Nanobase AI deploys and tunes vLLM on both EKS and AKS as part of production LLM serving builds.

Getting a pod scheduled on a GPU is the easy part

Installing the NVIDIA device plugin or the fuller GPU Operator and requesting a GPU resource limit in a pod spec is a well-documented first step on both EKS and AKS. The part that actually determines whether a vLLM deployment performs well in production is everything downstream of scheduling: how model weights reach the container, how tensor parallelism maps to node topology, and how autoscaling responds to LLM-specific load rather than generic CPU metrics. Teams that stop at "the pod is running" often discover the gaps only once real traffic hits the service.

EKS versus AKS: what actually differs

The NVIDIA GPU Operator itself behaves consistently across both, which is one reason Kubernetes is the common denominator teams standardize on for portability. Where the two clouds diverge is storage and networking primitives, which affects how model weights get onto the node and how multi-GPU pods communicate.

A minimal resource request for a vLLM pod

resources:
  limits:
    nvidia.com/gpu: 4
  requests:
    nvidia.com/gpu: 4

This example requests four GPUs for a single pod running a model split with tensor parallelism across four devices. The number here must match the --tensor-parallel-size value passed to vLLM at startup, and node affinity or anti-affinity rules should keep those four GPUs on the same physical instance so tensor-parallel communication stays on fast local interconnect rather than crossing nodes.

Operational details that separate a working deployment from a good one

None of the five items below show up in a basic tutorial, and skipping any one of them tends to surface as an intermittent production issue rather than a deployment failure.

1. Store model weights in object storage (S3 or Blob Storage) and pull them at container startup with a retry-aware init container, rather than baking multi-gigabyte weights into the image and slowing every deployment.
2. Set node affinity so multi-GPU pods land on instances with the GPUs physically connected via NVLink, not spread across nodes.
3. Scale horizontal pod autoscaling on request queue depth or GPU utilization from DCGM metrics, since CPU usage on an LLM serving pod barely moves regardless of load.
4. Configure liveness and readiness probes against vLLM's own health endpoint rather than a generic TCP check, so a pod stuck loading weights is not marked healthy prematurely.
5. Set pod disruption budgets carefully, since a GPU node replacement during a cluster upgrade can take longer to reschedule than a typical CPU workload.

Frequently asked questions

Do we need the full NVIDIA GPU Operator or is the device plugin enough?

The basic device plugin is enough for simple GPU scheduling, but the full GPU Operator additionally manages driver installation, DCGM monitoring, and MIG configuration, which matters for production clusters that need consistent driver versions and GPU utilization visibility across nodes.

How should tensor parallelism settings match Kubernetes GPU requests?

The GPU resource request in the pod spec must equal the tensor-parallel size passed to vLLM at startup, and node affinity rules should ensure all requested GPUs sit on the same physical instance, since cross-node tensor parallelism introduces communication latency that most interconnects outside InfiniBand handle poorly.

What metric should trigger autoscaling for a vLLM deployment?

Request queue depth or GPU memory and utilization pulled through NVIDIA DCGM exporters give a far more accurate scaling signal than CPU usage, since LLM inference is GPU-bound and CPU utilization on the serving pod stays low regardless of actual load.

Can the same vLLM deployment manifest work on both EKS and AKS?

Mostly yes for the container and Kubernetes resource definitions, but storage class names, node selector labels for GPU instance types, and ingress configuration typically need cloud-specific values, so a templated manifest with environment-specific overrides works better than a single hardcoded file.

How Nanobase AI helps

Nanobase AI, an NVIDIA Inception Program member, deploys and tunes vLLM on both EKS and AKS as part of production LLM serving builds, configuring node affinity, weight storage, and GPU-aware autoscaling rather than leaving a cluster at Kubernetes defaults. This work connects to broader Kubernetes GPU Operator versus Slurm decisions and to running GPU workloads across on-prem and cloud together.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.