KubeRay is a Kubernetes operator that manages Ray clusters as native custom resources, letting you define a RayCluster, RayJob, or RayService and have Kubernetes handle pod creation, autoscaling, and failure recovery for the underlying Ray head and worker nodes. You should use Ray on Kubernetes when your organization already standardizes infrastructure on Kubernetes and wants Ray-based training or serving pipelines to share the same cluster, GPU Operator drivers, monitoring stack, and access controls as other workloads, rather than operating a separate standalone Ray deployment. The RayJob resource is particularly useful for batch training or fine-tuning jobs that need to spin up a Ray cluster, run to completion, and tear down automatically, avoiding payment for idle GPU nodes between runs. RayService adds rolling updates and health checking for long-running inference deployments built on Ray Serve, valuable for production LLM serving that needs zero-downtime deploys. Teams running Ray purely for research on a static, dedicated cluster sometimes skip Kubernetes entirely and use Ray's own cluster launcher instead, since it is simpler when there is no need to share infrastructure. Nanobase AI, a Silicon Valley enterprise AI engineering company, deploys KubeRay when a customer's training and serving pipelines both need to live inside an existing Kubernetes GPU platform.
KubeRay's custom resources at a glance
| CRD | Purpose | Typical use |
|---|---|---|
| RayCluster | Long-running Ray head and worker pods | Persistent development or shared research cluster |
| RayJob | Spins up a Ray cluster, runs to completion, tears down | Batch training or fine-tuning runs |
| RayService | RayCluster plus rolling updates and health checks | Production inference serving with Ray Serve |
Choosing the right CRD for the workload shape, rather than defaulting to a single always-on RayCluster, is what keeps GPU spend proportional to actual usage.
RayJob for ephemeral training runs, step by step
- Define a RayJob manifest specifying the training entrypoint script and the resource requirements for head and worker pods.
- Apply it; KubeRay provisions a RayCluster sized to the job automatically rather than requiring a pre-existing standing cluster.
- The job runs to completion inside that ephemeral cluster, with logs and results persisted to shared storage or an object store.
- KubeRay tears down the RayCluster once the job finishes, releasing GPU nodes back to the pool instead of leaving them idle and billed.
This pattern avoids paying for idle GPU nodes between training runs, which matters especially on cloud infrastructure where idle GPU-hours are pure waste rather than sunk capital.
RayService for production inference
RayService adds rolling updates and health checking on top of a RayCluster running Ray Serve, letting a new model version deploy without dropping in-flight requests, since KubeRay keeps the previous version's pods running until the new version passes health checks. This is the piece that turns Ray Serve from a research serving tool into something operable in the way a Kubernetes Deployment is, with the same rolling-update guarantees production teams expect from any other service on the cluster.
KubeRay vs the plain Ray cluster launcher
Use KubeRay when your organization already standardizes on Kubernetes and wants Ray pipelines to share the same GPU Operator drivers, monitoring stack, and access controls as everything else running on the cluster. Teams running Ray purely for research on a static, dedicated set of machines with no need to share infrastructure with other workloads sometimes skip Kubernetes entirely and use Ray's own cluster launcher instead, since it is simpler to operate when there is genuinely nothing else on the hardware competing for the same resources or monitoring stack.
Frequently asked questions
Does KubeRay require the NVIDIA GPU Operator?
Yes, in practice, since KubeRay-managed pods still need GPU drivers, the container toolkit, and device plugin resources exposed the same way any other GPU pod on the cluster does; KubeRay itself only manages the Ray-specific orchestration layer on top. The GPU Operator typically handles that driver and toolkit installation automatically across the node pool, so KubeRay pods can request GPUs the same way any other Kubernetes workload does.
Can a RayJob request different GPU types for different stages?
Yes, a RayJob's worker groups can each specify different resource requirements, letting a multi-stage pipeline request, for example, smaller GPUs for data preprocessing and larger ones for training within the same job definition. This lets a single job definition mix GPU generations efficiently instead of provisioning the most expensive GPU type for every stage regardless of whether that stage actually benefits from the extra compute.
What happens if a RayService pod fails a health check mid-rollout?
KubeRay halts the rollout and keeps traffic on the previous healthy version rather than completing the update, preventing a bad deployment from taking down production inference traffic. Operators can then inspect the failing pod's logs to diagnose the issue and either fix the new version or roll back entirely, without ever having exposed live traffic to the broken deployment.
Is KubeRay suitable for Kueue-based fair-share GPU scheduling?
Yes, KubeRay integrates with queueing systems like Kueue so RayJob and RayCluster resources are subject to the same fair-share and quota policies as other Kubernetes GPU workloads on a shared cluster. This matters for multi-team clusters where Ray workloads need to compete fairly for GPU capacity alongside other Kubernetes-native training and inference jobs rather than bypassing the shared quota system entirely.
How Nanobase AI helps
Nanobase AI, an accepted member of the NVIDIA Inception Program, deploys KubeRay when a customer's training and serving pipelines both need to live inside an existing Kubernetes GPU platform, choosing RayJob or RayService based on whether the workload is ephemeral or long-running. For the underlying Ray scheduling model, see what is Ray and how does it schedule GPU jobs.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.