How do I run containers on Slurm with Pyxis and Enroot?

You run containers on Slurm with Pyxis and Enroot by installing Enroot, a lightweight container runtime built for HPC that unpacks OCI or Docker images into a rootless, chroot-like environment optimized for shared filesystems, alongside Pyxis, a Slurm plugin that adds container-aware flags directly to job submission commands. Once both are installed on compute nodes, a user launches a containerized job by pointing the submission command at a container image reference, and Pyxis handles pulling and caching the image through Enroot, mounting the container's filesystem, and injecting GPU device access, all without needing Docker or root privileges on the node, which matters in shared HPC environments where users should not have root. Enroot's image caching avoids re-pulling large container images, such as multi-gigabyte NGC PyTorch containers, for every job, significantly speeding up job startup on clusters running many short training or inference jobs. This combination gives Slurm clusters much of the container convenience that Kubernetes provides natively, without adopting Kubernetes itself, which is why large supercomputing centers and NVIDIA's own DGX SuperPOD reference designs ship Pyxis and Enroot as standard. Nanobase AI configures Pyxis and Enroot on every Slurm cluster it builds so customers can run standard NGC or custom containers without a separate orchestration layer.

Why HPC clusters chose Enroot over Docker

Docker requires a root-privileged daemon, which is a poor fit for shared HPC clusters where users submit jobs without root access and administrators do not want a persistent privileged service running on every compute node. Enroot solves this by unpacking OCI or Docker images directly into a rootless, chroot-like environment per job, while Pyxis adds the Slurm-side integration that makes launching a containerized job as simple as adding a flag to a normal job submission. This combination gives Slurm clusters most of the practical benefit of container-based workloads, reproducible environments, dependency isolation, without adopting a separate container orchestration platform.

Running a job in practice

srun --container-image=nvcr.io#nvidia/pytorch:24.05-py3 \
     --container-mounts=/data:/data \
     --gpus=8 \
     python train.py --config config.yaml

Pyxis intercepts the --container-image flag, hands the reference to Enroot, which pulls and caches the image locally if it is not already present, mounts the specified paths, injects GPU device access, and executes the command inside the container namespace. No separate container runtime daemon, registry authentication step, or orchestration layer is required beyond what is already configured on the node.

Caching behavior and its performance impact

Enroot's image cache should live on local NVMe rather than shared network storage, since caching to a shared filesystem defeats much of the performance benefit and adds unnecessary load to storage that other jobs depend on. Clusters running many short training or fine-tuning jobs against the same base image, a common NGC PyTorch or TensorRT-LLM container, see the largest startup-time improvement from this caching behavior.

Common pitfalls

Rootless execution means UID mapping between the host and container matters more than it does under Docker's root daemon model; a container image built assuming root ownership of certain paths can behave unexpectedly under Enroot's user namespace mapping, and this is worth testing before onboarding a new image broadly. ENROOT_RUNTIME_PATH should point to fast local storage, not a shared mount, for the reason above. Finally, GPU device injection depends on the NVIDIA Container Toolkit being correctly installed on the host even though Enroot itself is a separate runtime, so a node missing that toolkit will fail GPU-enabled container jobs with an error that looks unrelated to Enroot itself.

Why large HPC centers standardized on this stack

Pyxis and Enroot are the container approach NVIDIA ships as standard in its DGX SuperPOD reference designs, and many national supercomputing centers use the same combination, precisely because it avoids the operational overhead of running Kubernetes purely to get container isolation on top of an already-working Slurm deployment. Organizations already committed to Kubernetes for other reasons typically get equivalent container behavior through the NVIDIA Container Toolkit directly, without needing Pyxis at all.

Frequently asked questions

Do users need Docker installed to use Pyxis and Enroot?

No. Enroot reads OCI and Docker-format images but does not require a Docker daemon on the node. Users can reference standard Docker Hub or NGC image tags directly in their srun command without Docker being installed anywhere in the cluster.

Can Pyxis and Enroot run multi-node distributed training jobs?

Yes. Pyxis integrates with Slurm's normal multi-node job allocation, so a distributed training job spanning many nodes launches with the same container on every node through standard srun with --container-image, and NCCL communication between containers works the same as between bare processes on the underlying network.

Is Enroot a security risk since it runs without root?

Enroot is designed specifically to avoid the security exposure of a root-privileged container daemon, running instead within the submitting user's own privileges via user namespaces. This is generally considered safer for shared multi-tenant HPC systems than requiring root-level container tooling on every node.

How Nanobase AI helps

Nanobase AI configures Pyxis and Enroot on every Slurm cluster it builds, including cache placement and UID mapping validation, so customers run standard NGC or custom containers without adopting a separate orchestration layer they do not need.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.