Kubernetes GPU Operator vs Slurm: which should run your GPU cluster?

Use Slurm when your GPU cluster exists mainly to train models across many nodes, and use Kubernetes with the NVIDIA GPU Operator when it exists mainly to serve inference, APIs and other long-running services. Teams that do both usually run both: a Slurm partition for training and a Kubernetes GPU pool for serving, sharing storage, identity and monitoring. Picking one scheduler for every workload is the most expensive mistake in GPU clusters, because each tool is weak exactly where the other is strong.

What Kubernetes with the GPU Operator and Slurm each do

Kubernetes plus the NVIDIA GPU Operator

Kubernetes is a container orchestrator built for services that should never stop: it keeps replicas running, restarts failures, rolls out new versions and routes traffic. Out of the box it knows nothing about GPUs. The NVIDIA GPU Operator closes that gap by installing and managing the driver, container toolkit, device plugin, feature discovery, DCGM monitoring and MIG configuration as Kubernetes resources. After that a pod requests nvidia.com/gpu: 2 and lands on a node with two free GPUs.

That makes Kubernetes the natural home for vLLM, TensorRT-LLM and NVIDIA NIM inference servers, RAG and agent backends, notebooks and CI. What it lacks by default is a job queue, gang scheduling and fair-share accounting.

Slurm

Slurm is a batch scheduler and resource manager from SchedMD, used on many of the world's largest HPC systems. Users submit jobs with sbatch or srun, stating the nodes, GPUs and hours they need; the controller queues the job, applies priorities and fair-share, backfills small jobs around large ones, and starts every task on every allocated node at once. Accounting, reservations, preemption and wall-time limits are core features.

Slurm is built for work that starts, runs to completion and releases its resources, which is exactly the shape of a training run. It is a poor fit for a service that must answer requests indefinitely. Kubernetes schedules services that should never stop; Slurm schedules jobs that must start together and will finish.

Kubernetes GPU Operator vs Slurm: side-by-side comparison

The table compares both on the criteria that decide real deployments; "with add-ons" means it works, but you install and operate an extra component.

The comparison is not about which tool is better; it is about which gaps you are prepared to fill with add-ons and staff. Kubernetes needs three or four extra projects to match Slurm's training scheduler; Slurm needs a separate system to serve models in production.

When to choose Kubernetes, Slurm, or both

Choose Kubernetes with the GPU Operator when inference and services dominate GPU hours, when your organization already runs Kubernetes in production, and when you need autoscaling, namespaces per team and CI/CD integration. EKS, AKS and GKE all support it, so a hybrid strategy stays consistent. In the clusters Nanobase AI builds, a serving-heavy pool of RTX PRO 6000 or H100 PCIe nodes usually runs Kubernetes alone.

Choose Slurm when multi-node training dominates, when many researchers or teams need fair-share access to one pool, and when InfiniBand topology matters more than autoscaling. It also suits a small ops team that knows Linux better than Kubernetes, because a working Slurm cluster is a handful of config files rather than a platform. Fine-tuning a 70B model across 4 nodes of 8x H100 is a Slurm job, not a Kubernetes Deployment.

Running both: Slurm for training, Kubernetes for serving

The most common enterprise pattern as of 2026 splits the cluster into a training pool under Slurm and a serving pool under Kubernetes. Both share a parallel file system or object store for datasets and checkpoints, one container registry, one identity provider and one Prometheus and Grafana stack. A model trained under Slurm is exported, registered and deployed to Kubernetes without anyone copying files by hand.

Bridging options, from most to least integrated:

1. Slurm on Kubernetes: SchedMD's Slinky project (slurm-operator) and CoreWeave's SUNK run the Slurm daemons as pods; one physical layer, and researchers keep sbatch.
2. Kubernetes-native batch: Kueue for queues and quotas, Kubeflow Trainer or JobSet for multi-node jobs, Volcano or KAI Scheduler for gang scheduling; no second system, but Slurm is rebuilt piece by piece.
3. Containers in Slurm: Pyxis with Enroot, or Slurm's native OCI support, so the same image trains under Slurm and serves under Kubernetes.
4. Node hand-off: move whole nodes between pools during monthly rebalancing by reimaging or relabeling. Crude, but reliable and easy to audit.

Run both when training and serving are each large enough to justify their own scheduler; otherwise pick the tool that matches the dominant workload.

Key components of each stack

Kubernetes GPU stack

The GPU Operator installs these as DaemonSets and controllers, driven by one ClusterPolicy resource:

• NVIDIA driver: on the host image or as an operator-managed container; one version per node pool.
• NVIDIA Container Toolkit: configures containerd or CRI-O so containers receive /dev/nvidia* devices and matching libraries.
• Kubernetes device plugin: advertises nvidia.com/gpu and, with MIG, nvidia.com/mig-<profile> resources; also implements time-slicing and MPS.
• Node Feature Discovery and GPU Feature Discovery: label nodes with GPU product, driver and CUDA version, and MIG capability.
• DCGM exporter: Prometheus metrics for utilization, memory, power, temperature, NVLink and XID errors.
• MIG Manager: applies a MIG layout per node from a label such as nvidia.com/mig.config=all-3g.40gb.

Add the NVIDIA Network Operator for RDMA. A minimal GPU request:

apiVersion: v1
kind: Pod
metadata:
  name: vllm-h100
spec:
  nodeSelector:
    nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      resources:
        limits:
          nvidia.com/gpu: 2   # or nvidia.com/mig-3g.40gb: 1 in MIG mixed strategy

Slurm stack

A Slurm cluster is assembled from daemons and configuration files that must agree with each other:

• slurmctld: the controller, ideally with a backup and shared state directory; slurm.conf defines nodes, partitions and scheduler plugins.
• slurmd on each compute node, launching job steps through slurmstepd.
• slurmdbd with MariaDB: accounting for sacctmgr accounts, QOS and TRES limits; fair-share depends on it.
• cgroup.conf: ConstrainDevices=yes ensures a job sees only its allocated GPUs; ConstrainCores and ConstrainRAMSpace do the same for CPU and memory (cgroup v2 since Slurm 22.05).
• gres.conf: declares GPUs; AutoDetect=nvml reads count, type, NVLink and CPU affinity from the driver. See the Slurm GRES guide.

Common additions: Pyxis and Enroot for containers, slurmrestd for a REST API, Open OnDemand for a portal. A minimal GPU declaration:

# gres.conf on the compute nodes
AutoDetect=nvml
NodeName=gpu[01-16] Name=gpu Type=h100 File=/dev/nvidia[0-7]

# slurm.conf on the controller (abbreviated)
GresTypes=gpu
NodeName=gpu[01-16] Gres=gpu:h100:8 State=UNKNOWN
PartitionName=train Nodes=gpu[01-16] MaxTime=2-00:00:00 State=UP

On Kubernetes the operator assembles the GPU stack for you; on Slurm you assemble it, and gres.conf plus cgroup.conf decide whether jobs really get isolated GPUs.

Networking: InfiniBand, NVLink and NCCL

Inside a node, GPUs talk over NVLink and NVSwitch: 900 GB/s bidirectional per GPU on H100 and H200 SXM, 1.8 TB/s on B200, while PCIe cards such as the RTX PRO 6000 rely on PCIe Gen5 at about 128 GB/s bidirectional per x16 slot. Between nodes the standard is InfiniBand NDR at 400 Gb/s per port, one port per GPU in a rail-optimized design, or 3.2 Tb/s per 8-GPU node; XDR at 800 Gb/s ships with Blackwell-generation systems as of 2026. RoCE on Spectrum-X Ethernet is the alternative.

NCCL is the library every training framework uses for all-reduce and all-gather. It detects NVLink, InfiniBand and GPUDirect RDMA automatically, but silently falls back to TCP over the management network if the fabric is not visible to the process. Run a test job with NCCL_DEBUG=INFO, confirm it reports IB transports with GDRDMA, and record nccl-tests all-reduce numbers as an acceptance baseline.

On Slurm, jobs run on the host network, so InfiniBand works with nothing beyond topology.conf for switch-aware placement. On Kubernetes, pods live on an overlay network, so RDMA needs the Network Operator, Multus with SR-IOV or host-device interfaces, an RDMA device plugin, and pods that request an RDMA resource alongside GPUs. The fabric is identical on both; what differs is the work needed before a container can use it, and unfinished RDMA setup is the usual cause of "training is 3x slower on Kubernetes".

Operations checklist for a GPU cluster

The scheduler is rarely what breaks a GPU cluster; drivers, firmware, fabric and quota drift are. This checklist applies to both stacks.

1. Monitoring: ship DCGM exporter metrics to Prometheus and Grafana on every node. Alert on XID and ECC errors, thermal throttling and NVLink faults, and track allocated-but-idle GPUs, the largest hidden cost in most clusters.
2. Health gating: run dcgmi diag -r 2 after boot and maintenance; use Slurm's HealthCheckProgram with NHC, or Kubernetes node-problem-detector, to drain nodes with failing GPUs automatically.
3. Driver and CUDA compatibility: pin one driver version per node pool and check it against NVIDIA's CUDA compatibility guide. Images bring their own CUDA runtime, but the host driver must meet the minimum for that CUDA major version; minor-version compatibility and the forward-compatibility package cover most gaps.
4. Upgrades: on Kubernetes, upgrade the GPU Operator first, then roll drivers with driver.upgradePolicy and maxParallelUpgrades so nodes drain in groups. On Slurm, upgrade slurmdbd, then slurmctld, then slurmd, inside the version window SchedMD documents. Canary both on a small node pool.
5. Firmware: track GPU VBIOS, ConnectX NIC firmware and switch OS versions; use Slurm maintenance reservations or Kubernetes cordons for windows.
6. Fairness review: report queue wait time and GPU-hours per project monthly; enforce limits with QOS and GrpTRES on Slurm or Kueue ClusterQueue quotas on Kubernetes.
7. Storage: verify every training node writes checkpoints at full speed; a full 405B checkpoint with optimizer state is several terabytes. Cache datasets on local NVMe.
8. Runbooks: how to drain a node, replace a GPU, roll back a driver and restore the Slurm database or etcd.

Automate compatibility and health checks; that removes most GPU cluster incidents before the scheduler ever sees them. For sizing the nodes themselves, see how many GPUs you need for 70B, 405B and DeepSeek R1 and the on-premise LLM deployment guide.

Frequently asked questions

Can Kubernetes run multi-node distributed training?

Yes, but not with the stock scheduler alone. You need a job API such as Kubeflow Trainer or JobSet to launch ranks across pods, a gang scheduler such as Volcano, KAI Scheduler or Kueue, and RDMA networking so NCCL uses InfiniBand rather than TCP. With those in place, throughput matches Slurm on the same hardware; without them training can be several times slower.

Can Slurm serve inference endpoints in production?

A Slurm job can run vLLM and expose a port, but Slurm has no service discovery, load balancing, health-based restarts or rolling updates, and every job has a wall-time limit. Some sites run long "forever jobs" for internal demos. For anything customer-facing, serve on Kubernetes or a VM fleet and let Slurm focus on training.

Is the NVIDIA GPU Operator required to use GPUs on Kubernetes?

No. You can install the driver, container toolkit and device plugin by hand, and GKE, EKS and AKS ship driver-enabled node images. The operator is still worth using because it manages upgrades, MIG layouts, DCGM monitoring and feature discovery consistently. On managed clouds you typically set driver.enabled=false so the cloud keeps the driver and the operator manages the rest.

How does MIG work on Kubernetes versus Slurm?

MIG partitions an H100, H200 or B200 into up to seven isolated instances with dedicated memory and compute. On Kubernetes, MIG Manager applies a profile per node and the device plugin advertises each slice as a resource such as nvidia.com/mig-1g.10gb. On Slurm, AutoDetect=nvml exposes MIG instances as individual GPUs that jobs request normally. Kubernetes makes changing layouts easier; Slurm makes accounting easier.

Can I run Slurm inside Kubernetes?

Yes, and as of 2026 it is a supported pattern. SchedMD's Slinky project provides a Slurm operator that runs the controller and node daemons as pods, and GPU clouds offer similar stacks such as CoreWeave's SUNK. Researchers keep sbatch and fair-share, the platform team keeps one physical layer. The trade-off is two schedulers to understand when something goes wrong.

What is the difference between time-slicing and MIG?

Time-slicing lets several pods share one GPU by switching between them; it adds no memory isolation, so one tenant can exhaust memory for the others, and it suits notebooks and light inference. MIG carves the hardware into isolated instances with guaranteed memory and compute, the right choice for multi-tenant production inference. MPS sits between them: lower switching overhead, still no hard memory limits.

How Nanobase AI can help

Nanobase AI designs, installs and operates GPU clusters end to end: hardware sizing for H100, H200, B200 and RTX PRO nodes, InfiniBand and NVLink fabric planning, Kubernetes with the NVIDIA GPU Operator for serving, Slurm for training, and the bridges between them. We configure MIG layouts, DCGM monitoring, fair-share quotas and driver upgrade pipelines, then hand over runbooks your team can run without us. As a Silicon Valley company and an NVIDIA Inception Program member, we also deploy the inference layer on top, from vLLM and TensorRT-LLM to NVIDIA NIM, on-premise or hybrid with AWS, Azure and Google Cloud. See our solutions overview for the full scope.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.