You upgrade NVIDIA drivers on a GPU cluster without downtime by rolling the upgrade through the cluster node by node or rack by rack, draining and cordoning each node before touching its driver so no job is ever running on a node mid-upgrade. On Kubernetes, the GPU Operator supports rolling upgrades where you cordon a node, wait for its pods to be rescheduled elsewhere, let the operator's driver component update and reload, then uncordon the node and move to the next one, coordinated by a maintenance controller if you want it fully automated. On Slurm clusters, the equivalent approach is setting a node to a drain state so the scheduler stops assigning new jobs while letting currently running jobs finish, then applying the update once the node is idle. Capacity planning matters here since usable GPU count temporarily drops during the rolling window, so schedule upgrades during lower-demand periods and keep enough headroom that draining a few nodes at a time does not create a queuing backlog. Always test the new driver version against a representative workload on one canary node before rolling it fleet-wide. Nanobase AI performs rolling driver upgrades as part of its ongoing GPU cluster operations service.
Rolling upgrade runbook for Kubernetes
- Pin the target driver version in the GPU Operator's ClusterPolicy rather than tracking a moving default.
- Cordon one node so the scheduler stops placing new pods on it, then drain it to reschedule existing pods elsewhere.
- Let the GPU Operator's driver component update and reload on that node, watching for the validator pod to report healthy again.
- Run a quick smoke test, a pod requesting a GPU and executing
nvidia-smi, before uncordoning. - Uncordon the node and move to the next, using a maintenance controller to automate the sequence if the fleet is large enough to make manual coordination impractical.
Never update more than a small batch of nodes at once during the initial rollout, since a bad driver build affecting every node simultaneously turns a routine upgrade into a fleet-wide outage.
Rolling upgrade runbook for Slurm
| Step | Action |
|---|---|
| 1 | Set the target node to a drain state so the scheduler stops assigning new jobs to it |
| 2 | Wait for currently running jobs on that node to finish naturally |
| 3 | Apply the driver update once the node reports idle |
| 4 | Run a validation job requesting that node specifically before returning it to service |
| 5 | Remove the drain state and repeat for the next node or batch |
Slurm's drain state is the direct equivalent of Kubernetes cordon and drain, and the same batch-by-batch discipline applies: update a small number of nodes, validate, then continue rather than draining the entire partition simultaneously.
Canary and rollback strategy
Test the new driver version against a representative workload on one canary node before it touches any node serving real traffic or running a real training job, since a driver regression specific to your exact workload pattern is exactly the kind of issue a generic release note will not mention. Keep the previous driver container image tag or package version readily available so a rollback is a configuration change, pointing the ClusterPolicy or configuration management tool back at the prior version, rather than a manual reinstall scrambled together during an incident.
Capacity planning during the rolling window
Usable GPU count temporarily drops during a rolling upgrade window, so schedule the maintenance during lower-demand periods and keep enough spare capacity that draining a few nodes at a time does not create a queuing backlog for other teams. A cluster running at or near full utilization has no slack to absorb a rolling upgrade gracefully, which is a strong argument for building headroom into capacity planning specifically to accommodate routine maintenance windows rather than treating every GPU as permanently allocated.
Frequently asked questions
How many nodes should be upgraded at once in a rolling window?
Start with a single canary node, then expand to small batches, typically a handful of nodes at a time, once the canary confirms the new version behaves correctly. The right batch size scales with cluster size and how much spare capacity is available to absorb draining.
What is the biggest risk in a driver rolling upgrade?
Skipping the canary step and rolling the new driver to every node at once, since an undetected regression then affects the entire fleet simultaneously instead of being caught and rolled back after impacting only one node. This is why the canary step exists at all; the small amount of extra time it takes to validate one node first is trivial compared to the cost of diagnosing and recovering from a fleet-wide regression.
Does a driver upgrade always require draining running jobs first?
Yes, in practice, since the driver container reload disrupts any process currently using the GPU. Cordoning or draining ahead of time is what prevents an in-progress training job or inference request from failing mid-execution. Skipping this step to save time is a common cause of mid-upgrade incidents, since a job that loses its GPU mid-execution typically has to restart from its last checkpoint rather than resuming cleanly.
Can driver upgrades be fully automated end to end?
Largely yes on Kubernetes using a maintenance controller that handles cordon, drain, upgrade, validate, and uncordon in sequence, though most teams still keep a canary and manual approval gate before the automation proceeds to the next batch. That manual gate is less about distrust of automation and more about catching workload-specific regressions that automated health checks are not designed to detect on their own.
How Nanobase AI helps
Nanobase AI performs rolling driver upgrades as part of its ongoing GPU cluster operations service, canary-testing every new version against representative workloads before it reaches production nodes. For choosing the right version in the first place, see which NVIDIA driver and CUDA version to install.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.