How do I checkpoint and resume failed GPU training jobs?

You checkpoint and resume failed GPU training jobs by saving model weights, optimizer state, and training step metadata to durable shared storage at a regular interval, then configuring your training script or orchestration layer to detect a restart and automatically resume from the most recent complete checkpoint rather than starting over. Most training frameworks, including PyTorch's distributed checkpoint utilities and higher-level libraries like DeepSpeed or Megatron, support asynchronous checkpoint writing so saving state does not stall GPU compute for the full duration of the write, which matters because checkpoints for large models can be tens or hundreds of gigabytes and a synchronous save at that size wastes significant GPU-hours. Checkpoint frequency is a trade-off: too frequent and you spend meaningful time and storage bandwidth writing state, too infrequent and a failure late in an interval costs more recomputed work, with many large training runs settling on intervals between fifteen minutes and a few hours depending on model size and failure rate. On Kubernetes, a job controller such as Kubeflow's training operator or a custom restart policy handles automatic resubmission after a pod failure, while on Slurm a requeue flag on the job submission achieves the same effect. Nanobase AI, headquartered in Silicon Valley, builds fault-tolerant checkpointing into every multi-node training pipeline it sets up.

The checkpoint interval is an optimization, not a default

Many teams pick a checkpoint interval, commonly a round number like every hour, without connecting it to the two variables that should actually drive the choice: how long a checkpoint write costs in stalled or reduced-throughput GPU time, and how often failures actually occur on the cluster. The right interval minimizes total wasted compute, which is checkpoint overhead plus expected recompute cost from failures, not a fixed calendar interval chosen out of habit.

The trade-off in one table

As cluster size grows, expected failure frequency across the whole job rises even if any single node is reliable, because a distributed job fails whenever any one of its many nodes fails. This is why large-scale training runs, spanning hundreds of GPUs, often settle on shorter intervals than a single-node job would need, despite each node individually being no less reliable.

Making checkpoint writes cheap enough to run often

Asynchronous checkpointing is what makes frequent saving practical at all: frameworks including PyTorch's distributed checkpoint utilities, DeepSpeed, and Megatron support writing checkpoint state to storage in the background while GPU compute continues on subsequent steps, rather than blocking training for the full duration of a write that can take minutes for a large model's combined weights and optimizer state. Without asynchronous writing, checkpoint size directly caps how often you can afford to save without meaningfully slowing training.

Checkpoint strategies compared

1. Synchronous local checkpoint — fast to write, but training blocks completely during the save and the checkpoint is lost if the node itself fails, offering no real fault tolerance against node-level failure.
2. Asynchronous checkpoint to local disk — training continues during the write, but still vulnerable to node failure since the checkpoint never leaves the failed node.
3. Asynchronous checkpoint to shared storage — the standard approach for multi-node training, tolerating node failure since the checkpoint survives on independent shared storage, at the cost of network bandwidth to that storage tier.
4. In-memory replicated checkpointing — replicates state across peer GPUs' memory for very fast recovery from single-node failure without a storage round trip, an emerging pattern for very large training runs willing to accept the added engineering complexity.

Automating the resume path

On Kubernetes, a job controller such as Kubeflow's training operator handles automatic resubmission after a pod failure, restarting the job so it can detect and load the latest checkpoint on startup. On Slurm, a requeue flag on the job submission achieves the equivalent behavior, returning a failed job to the queue rather than requiring manual resubmission. Either way, the training script itself needs to reliably detect a fresh start versus a resume and locate the correct checkpoint automatically, since a manual, error-prone resume process undermines the value of automated requeueing.

Frequently asked questions

Does checkpoint frequency affect final model quality?

No, checkpoint frequency affects fault tolerance and wasted compute on failure, not the model's final quality, assuming checkpoints correctly capture full training state including optimizer state and learning rate schedule position rather than weights alone.

How large are checkpoints for large language models?

Checkpoint size scales with parameter count and optimizer choice; a 70B-parameter model's full training state, including optimizer state such as Adam's momentum terms, is commonly several times larger than the roughly 140 GB of FP16 weights alone, which is why async writing to fast shared storage matters at this scale.

Should checkpoint frequency change over the course of a long training run?

Some teams checkpoint more frequently early in a run when hyperparameter issues are more likely to require a rollback, then relax frequency once training has stabilized, though this adds process complexity that should be weighed against the simplicity of one consistent interval.

How Nanobase AI helps

Nanobase AI, headquartered in Silicon Valley, builds fault-tolerant checkpointing into every multi-node training pipeline it sets up, sizing checkpoint interval against actual cluster failure rates and write overhead rather than a default value, and wiring automatic resume through Kubernetes or Slurm job controllers.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.