Ray is an open-source distributed computing framework that lets Python code scale from a laptop to a large cluster with minimal changes, and it has become a common backbone for both LLM training pipelines and inference serving because of its native support for GPU-aware task and actor scheduling. Ray's scheduler tracks custom resources, including GPU count and even specific GPU types, per node, and a task or actor requesting GPUs is only placed on a node with enough free capacity, with Ray also setting the visible-devices environment variable automatically so a process only sees the GPUs it was assigned. Ray Train and Ray Serve build on this core scheduler to handle distributed training loops and model serving respectively, while Ray Data handles GPU-accelerated preprocessing pipelines that feed training jobs. Because Ray's scheduler is topology-aware to a degree, it can also respect placement group constraints that keep tightly coupled GPU workers on the same node or rack for lower-latency collective communication. This makes Ray popular for fine-tuning and RLHF pipelines that combine multiple heterogeneous stages, such as generation, reward scoring, and policy update, each needing a different GPU count. Nanobase AI builds Ray-based training and inference pipelines for customers running multi-stage AI workloads.
The Ray ecosystem at a glance
| Component | Role |
|---|---|
| Ray Core | Task and actor scheduling, including GPU-aware resource tracking |
| Ray Train | Distributed training loops across multiple GPU workers |
| Ray Serve | Model serving with autoscaling and request routing |
| Ray Data | GPU-accelerated data preprocessing pipelines feeding training jobs |
| Ray Tune | Hyperparameter search across many parallel GPU trials |
Ray's core scheduling model, tracking GPU count and type as first-class custom resources per node, is what lets these higher-level libraries share one cluster instead of each needing its own resource manager.
How the scheduler actually places GPU work
Ray's scheduler tracks GPU count and, when specified, GPU type per node, and a task or actor requesting GPUs is only placed on a node with enough free capacity to satisfy that request. Ray also sets the visible-devices environment variable automatically for a scheduled process, so a task only sees the specific GPUs it was assigned rather than every GPU physically present on the node, preventing accidental resource collisions between unrelated tasks sharing a node.
Placement groups and topology awareness
Placement groups let a job request that several actors or tasks land close together, for example all on the same node or rack, rather than scattered arbitrarily across the cluster. This matters directly for tightly coupled GPU workers that communicate frequently during training, since placing them on the same node keeps that communication on fast NVLink rather than forcing it across a slower network link. A STRICT_PACK placement strategy forces all bundles onto the fewest nodes possible, while SPREAD does the opposite for workloads that benefit from fault isolation instead of proximity.
Why multi-stage pipelines favor Ray
Fine-tuning and RLHF pipelines commonly combine several heterogeneous stages, generation, reward scoring, and policy update, each needing a different GPU count and sometimes a different GPU type. Ray's actor model lets each stage run as its own long-lived process with its own resource request, coordinated through the same cluster and scheduler rather than stitching together separate job systems for each stage. This heterogeneous multi-stage capability is Ray's clearest advantage over a simpler batch scheduler, which typically assumes each job has one uniform resource shape rather than several cooperating stages with different shapes.
Frequently asked questions
Does Ray support fractional GPU requests?
Yes, a task or actor can request a fraction of a GPU, such as 0.5, letting multiple lightweight tasks share one physical GPU, though this is a software-level scheduling hint rather than hardware-enforced isolation like MIG. This makes it a good fit for lightweight inference or preprocessing tasks that do not need a full GPU's memory or compute, but teams needing strict isolation between tenants should look at MIG instead.
Can Ray schedule across GPU types in one cluster?
Yes, by specifying a required GPU type as a custom resource alongside plain GPU count, letting a heterogeneous cluster route different tasks to the GPU generation each one actually needs. This is particularly useful for pipelines where different stages have very different compute requirements, such as routing lightweight preprocessing to older GPUs while reserving newer, more expensive GPUs exclusively for the training or inference stage that actually needs them.
Does Ray replace Kubernetes for GPU scheduling?
Not typically; many production deployments run Ray on top of Kubernetes through KubeRay, letting Ray handle fine-grained task and actor placement while Kubernetes manages the underlying node pool, driver stack, and cluster lifecycle. This division of responsibility means teams keep their existing Kubernetes tooling for node provisioning and monitoring while still getting Ray's finer-grained scheduling for individual tasks and actors within that cluster.
Is Ray suitable for large-scale pretraining, the way Slurm is?
Ray is used for training pipelines and fine-tuning widely, but the largest frontier-scale pretraining runs still commonly favor Slurm's mature gang scheduling; Ray's strength is heterogeneous, multi-stage pipelines rather than the largest single homogeneous training job. Teams running the largest homogeneous pretraining jobs typically stick with Slurm's battle-tested gang scheduling, while reserving Ray for the more heterogeneous fine-tuning, RLHF, and multi-stage pipeline work where its actor model provides a clearer advantage.
How Nanobase AI helps
Nanobase AI builds Ray-based training and inference pipelines for customers running multi-stage AI workloads, from placement group design for tightly coupled GPU workers to autoscaling configuration for bursty inference traffic. For running Ray on a Kubernetes-managed cluster, see what is KubeRay and when to use it.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.