Setting up Slurm for a GPU cluster starts with installing the controller, database, and compute node daemons across your nodes, then configuring the main Slurm configuration and its generic resource file so Slurm recognizes GPUs as a GRES resource that jobs can request explicitly, for example specifying a GPU type and count in a submission flag. The generic resource file on each compute node maps physical GPU device files to the GRES name and, on multi-GPU nodes, should also encode NUMA and PCIe topology information so Slurm places jobs on GPUs close to the CPU cores and network interfaces they will use, which matters for distributed training performance. You will also want to enable cgroup-based GPU isolation so a job cannot see or access GPUs it was not allocated, configure accounting through the Slurm database daemon so GPU-hours are tracked per user and account for fair-share scheduling, and set up Pyxis and Enroot if workloads run in containers rather than as bare processes. Test the setup with a simple multi-node job requesting specific GPU counts before onboarding real training workloads, and validate GPU-to-network affinity with nccl-tests once nodes are online. Nanobase AI, an NVIDIA Inception Program member, configures and hardens Slurm GPU clusters for enterprise customers running large-scale training.

Affinity matters more than raw GPU count

Many first-time Slurm deployments get GPUs recognized as a schedulable resource and stop there, treating every GPU on a node as interchangeable. On multi-GPU nodes, which GPU a job lands on relative to its assigned CPU cores and network interface directly affects distributed training performance, since PCIe topology determines whether a GPU shares a root complex with the NIC it needs for RDMA traffic. A gres.conf entry that only maps a device file to a GRES name without encoding this locality lets Slurm place jobs correctly by count but suboptimally by topology, which shows up as inconsistent step times across otherwise identical job submissions.

A topology-aware gres.conf entry

AutoDetect=nvml
Name=gpu Type=h100 File=/dev/nvidia0 Cores=0-15
Name=gpu Type=h100 File=/dev/nvidia1 Cores=0-15
Name=gpu Type=h100 File=/dev/nvidia2 Cores=16-31
Name=gpu Type=h100 File=/dev/nvidia3 Cores=16-31

The Cores field binds each GPU to the NUMA-local CPU core range that shares its PCIe root complex, so Slurm's CPU-to-GPU binding for a job actually reflects hardware locality rather than an arbitrary assignment. On 8-GPU nodes with a more complex PCIe switch topology, slurm.conf's topology.conf should additionally describe switch-level distances between nodes so multi-node job placement favors nodes that are close on the network fabric.

Deployment checklist

  1. Install slurmctld on the controller, slurmdbd backed by a MySQL or MariaDB database for accounting, and slurmd on every compute node.
  2. Configure slurm.conf with node definitions, partition definitions, and GresTypes=gpu, matching the GRES entries defined per node.
  3. Write gres.conf per node with device files, GPU type, and core affinity as shown above; use AutoDetect=nvml where hardware is uniform to reduce manual entry error.
  4. Enable cgroup based GPU isolation through ConstrainDevices=yes in cgroup.conf so a job cannot see GPUs it was not allocated, which matters for both security and clean error attribution.
  5. Bring up slurmdbd accounting and create accounts and associations for fair-share and quota policy before onboarding real workloads.
  6. Install Pyxis and Enroot if workloads run as containers rather than bare processes, which is standard for most modern ML training stacks.
  7. Submit a multi-node test job requesting a specific GPU type and count, then run nccl-tests across the allocation to validate GPU-to-network affinity before declaring the cluster production-ready.

Common configuration mistakes

A frequent early mistake is leaving SelectType=select/linear instead of select/cons_tres, which prevents Slurm from doing fine-grained GPU allocation and forces whole-node scheduling even for single-GPU jobs, wasting capacity on any cluster running mixed job sizes. Another common gap is omitting AccountingStorageTRES=gres/gpu from slurm.conf, which silently disables GPU-hour tracking in the accounting database even though jobs run correctly, undermining fair-share and cost reporting later. Finally, teams sometimes configure GRES without validating the actual PCIe topology against the vendor's node layout, assuming a symmetric design that may not match the specific server SKU deployed.

Frequently asked questions

Does Slurm support heterogeneous GPU types in one cluster?

Yes, through the Type field in gres.conf, which lets a partition or job request a specific GPU generation, such as H100 versus A100, rather than treating all GPU resources as identical. This is standard for clusters that grew across multiple hardware generations.

How does Slurm differ from Kubernetes for GPU scheduling?

Slurm was purpose-built for HPC batch scheduling with native fair-share, topology awareness, and GRES resource types, while Kubernetes schedules primarily on current resource availability and needs additional layers like Kueue for comparable batch fairness. See the detailed comparison for a fuller decision framework.

Can Slurm and Kubernetes run on the same physical cluster?

Yes, some organizations partition nodes between the two, or use Slurm as the primary scheduler with Kubernetes running inside Slurm-allocated resources for specific workloads, though this adds operational complexity that should be justified by a genuine mixed-workload requirement.

How Nanobase AI helps

Nanobase AI, an NVIDIA Inception Program member, configures Slurm clusters with topology-aware GRES mapping, accounting, and container support built in from the initial deployment rather than retrofitted after performance problems appear. We validate every cluster against nccl-tests and DCGM diagnostics before handing it over for production training.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.