The NVIDIA GPU Operator is a Kubernetes operator that automates deployment and lifecycle management of the software a node needs to expose and run NVIDIA GPUs, including the driver, container toolkit, device plugin, DCGM exporter, and validation containers. It packages these pieces as containers and installs them through custom resource definitions, so an administrator applies one Helm chart instead of hand-installing drivers on every node. The operator detects GPU hardware automatically, handles driver upgrades and node reboots in a controlled rolling fashion, and exposes GPU health and utilization metrics for Prometheus scraping. It also configures MIG partitioning and time-slicing policies through a single ConfigMap, which matters once a team mixes A100, H100, or H200 nodes in one pool. Without it, teams typically spend days per node reconciling driver, CUDA, and container runtime versions, and small mismatches cause CUDA initialization failures that are hard to trace. Most production Kubernetes GPU clusters on bare metal or cloud VMs with passthrough GPUs run this operator today. Nanobase AI, a Silicon Valley enterprise AI engineering company, deploys and tunes the GPU Operator as part of every Kubernetes-based GPU cluster it builds.
How the operator's components fit together
The GPU Operator is not a single binary; it is a set of controllers and DaemonSets that each own one piece of the GPU stack and reconcile it independently. Node Feature Discovery labels every node with the hardware it detects, the driver container installs a kernel-matched NVIDIA driver inside a container rather than on the host filesystem, the NVIDIA Container Toolkit exposes that driver into workload containers, the device plugin advertises nvidia.com/gpu as an allocatable Kubernetes resource, and dcgm-exporter ships utilization and health metrics to Prometheus. A validator pod runs after each piece starts and blocks scheduling on a node until the whole chain reports healthy.
| Component | Role |
|---|---|
| Node Feature Discovery | Labels nodes with GPU model, driver readiness, PCIe topology |
| Driver container | Installs and loads the NVIDIA kernel driver inside a container |
| Container Toolkit | Injects driver libraries and device nodes into workload containers |
| Device plugin | Advertises GPUs as a schedulable Kubernetes resource |
| dcgm-exporter | Exposes GPU health and utilization metrics to Prometheus |
| MIG Manager | Applies MIG partition profiles when configured |
| GPU Feature Discovery | Adds fine-grained labels such as MIG capability and compute mode |
Treating the operator as one reconciliation loop across these seven pieces, rather than seven separate installs, is what makes the deployment model reliable at fleet scale. A change to any one component, such as a driver version bump, flows through the same custom resource and rolls out the same way on every node. For the install sequence itself, see how to install the NVIDIA GPU Operator on Kubernetes.
Why the driver runs in a container, not on the host
Running the driver as a container rather than a host package is the operator's most consequential design choice. It decouples the driver version from the base OS image, so a fleet running Ubuntu 22.04 and one running a minimal container-optimized OS both get the identical driver build pulled from the same registry. It also makes rollback simple: reverting a driver is a matter of pointing the ClusterPolicy at a previous image tag rather than uninstalling and reinstalling host packages node by node.
The trade-off is that the driver container needs privileged access and a working kernel module build environment on first boot. Node images with a hardened kernel or a restrictive security policy sometimes need extra configuration before the driver container comes up cleanly. That first-boot compile step, not the Helm install itself, is the single most common source of a stuck rollout on a brand-new node image.
When you skip the GPU Operator's driver component
Several managed Kubernetes GPU node pools, including GKE nodes on Container-Optimized OS and some EKS AMI-based node groups, ship the NVIDIA driver pre-installed at the OS image layer. There you disable the operator's own driver piece and let it manage only the toolkit, device plugin, and monitoring components on top of the pre-baked driver. Running the operator's driver container over an already-installed host driver causes conflicts, typically a device plugin reporting zero allocatable GPUs or a validator pod stuck in a crash loop. The real configuration question on managed cloud Kubernetes is which components to enable, not whether to use the operator at all.
Operator versioning and upgrade cadence
GPU Operator releases are tested against specific driver, CUDA, and Kubernetes API combinations, and NVIDIA publishes a compatibility matrix for each release. Upgrading the operator without checking that matrix is a common cause of fleet-wide outages, since a newer operator version can pull a driver container incompatible with an older CUDA runtime baked into existing application images. Pinning the operator's Helm chart version explicitly in your GitOps configuration, rather than tracking the latest release, is the single change that prevents most operator-upgrade incidents. Stage every upgrade on a small canary node pool before rolling it across production.
Frequently asked questions
Does the GPU Operator work on cloud-managed Kubernetes like EKS and GKE?
Yes, but typically with the driver component disabled since managed node images often ship a pre-installed driver. You still get the toolkit, device plugin, and monitoring pieces, letting cloud and on-premise clusters share the same GPU Operator configuration for everything except driver management.
Does the GPU Operator support ARM nodes?
Support depends on the specific operator release and driver branch. NVIDIA has extended coverage to Grace-based and other ARM platforms in recent releases, so check the current compatibility matrix for your operator version before assuming parity with x86 nodes, especially for less common component combinations.
Can I run the GPU Operator without MIG or time-slicing configured?
Yes. MIG and time-slicing are optional ConfigMap settings layered on top of the base install. Most clusters start with neither enabled and add GPU sharing only once a specific workload mix, such as many small inference services, justifies the added configuration complexity.
What happens if the driver container fails to build the kernel module?
The validator pod blocks and the node stays unschedulable for GPU workloads until the issue resolves. Common causes are a kernel version without matching headers available, or a restrictive security policy preventing the privileged container needed for module loading on first boot.
How Nanobase AI helps
Nanobase AI, a Silicon Valley enterprise AI engineering company, configures the GPU Operator's component set to match each cluster's actual environment, whether that means full driver management on bare metal or a toolkit-and-monitoring-only setup layered on a cloud provider's pre-installed driver. That includes version-pinning the operator against a tested compatibility matrix and validating the rollout on our platform before it touches production nodes. For teams comparing this against a Slurm-based stack, see Kubernetes GPU Operator vs Slurm.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.