Spot or preemptible GPU instances can be used for LLM inference, but only for workloads that can tolerate interruption, since these instances can be reclaimed by the cloud provider with little or no notice in exchange for a substantial discount, often 60 to 90 percent off on-demand pricing depending on GPU type and region. They work well for batch inference jobs, asynchronous processing queues, and non-latency-critical workloads where a request can be retried on another instance if one gets preempted, and are commonly paired with a Kubernetes autoscaler that gracefully drains and reschedules pods when a preemption notice arrives. They are generally a poor fit for latency sensitive, synchronous production inference serving live user traffic, since an unexpected preemption can cause request failures or timeouts that directly affect user experience. A hybrid pattern many teams use is running a stable on-demand or reserved baseline for guaranteed capacity while adding spot instances only for overflow batch work or model evaluation tasks. Availability of spot H100 or H200 capacity also varies significantly by region and time, so relying on it as primary capacity carries real risk. Nanobase AI designs inference architectures that use spot capacity where it is safe while protecting production service levels with reserved GPU baselines.
The discount is real; so is the risk it trades against
Spot and preemptible GPU instances offer a substantial discount off on-demand pricing, often 60 to 90 percent depending on GPU type and region, in exchange for the cloud provider's right to reclaim the instance with little or no notice. The discount is only a good deal if the workload can absorb an interruption without a user-visible failure, which is a workload design question, not a pricing question. Treating spot capacity as simply "cheaper GPUs" without building interruption handling around it is how a cost optimization turns into a reliability incident.
The workloads that fit, and the ones that do not
| Workload | Spot suitability | Reasoning |
|---|---|---|
| Batch inference on a job queue | Good fit | A preempted job can be requeued and retried elsewhere |
| Asynchronous processing pipelines | Good fit | No user is waiting synchronously for the response |
| Model evaluation or benchmarking | Good fit | Interruption delays results but causes no user impact |
| Synchronous, latency-sensitive production serving | Poor fit | A mid-request preemption directly causes a failed or slow user request |
The dividing line is whether a human or downstream system is waiting synchronously for the result: if yes, spot capacity should not be the only path serving that request.
Handling preemption gracefully in Kubernetes
A safe spot GPU pattern in Kubernetes relies on a preemption notice, typically delivered with a short warning window before reclamation, triggering a graceful pod drain rather than an abrupt kill. The cluster autoscaler or a dedicated tool needs to watch for this signal, cordon the affected node, and reschedule in-flight work onto other capacity before the instance is actually reclaimed. A PodDisruptionBudget combined with a preStop hook that allows in-flight requests to complete, or fails them back to a retry queue cleanly, is the standard mechanism:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: inference-spot-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: llm-inference-spotThis ensures a minimum number of pods stay available even as individual spot nodes are reclaimed and replaced, keeping aggregate capacity stable even though individual instances are not.
The hybrid baseline pattern most teams converge on
Rather than choosing spot or on-demand exclusively, the pattern that balances cost and reliability best runs a stable on-demand or reserved baseline sized to guaranteed minimum traffic, with spot instances added only for overflow batch work or additional inference capacity above that baseline. This means production traffic never depends entirely on spot availability, while the cost savings still apply to whatever portion of capacity spot can safely absorb. Sizing the on-demand baseline correctly, neither so large that spot's cost advantage goes unused nor so small that a spot reclamation event degrades service, takes iteration against real traffic patterns rather than a one-time estimate.
Frequently asked questions
How much notice do spot or preemptible instances typically give before reclamation?
It varies by cloud provider, typically ranging from under a minute to a couple of minutes, which is enough time for a well-configured Kubernetes pod to drain gracefully but not enough time for a slow or unprepared shutdown process to complete safely.
Can spot GPU capacity be relied on for a majority of production inference traffic?
Not for latency-sensitive, synchronous traffic; relying on spot for the majority of that traffic risks noticeable service degradation during reclamation events. It works better as supplemental capacity above a stable, non-preemptible baseline.
Does spot availability vary significantly by GPU type?
Yes, spot availability for high-demand GPUs like H100 or H200 tends to be less predictable than for older or less contested GPU types, since spot capacity comes from unused on-demand inventory that shrinks during periods of high overall demand.
Is there a way to test spot preemption handling before relying on it in production?
Yes, most cloud providers offer a way to simulate or manually trigger a preemption event for testing, which should be used to validate the graceful drain and retry logic works as expected before depending on it for real traffic.
How Nanobase AI helps
Nanobase AI designs inference architectures that use spot capacity where it is genuinely safe while protecting production service levels with a reserved GPU baseline, building the graceful preemption handling that makes the cost savings real rather than risky. See our Kubernetes GPU Operator vs Slurm guide for the broader scheduling foundation this pattern relies on.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.