How do I schedule GPU jobs fairly across teams?

You schedule GPU jobs fairly across teams by combining a quota or fair-share policy at the scheduler level with visibility into actual usage, so no single team can monopolize shared capacity while legitimate bursts of demand are still accommodated. On Slurm, fair-share scheduling is built in through its multifactor priority mechanism, which lowers a user's or account's job priority temporarily after they consume more than their allocated share of GPU-hours, automatically rebalancing access over time without an administrator manually intervening. On Kubernetes, achieving the same effect typically requires an additional layer such as Kueue or Run:ai on top of the default scheduler, since Kubernetes alone schedules by resource availability rather than historical usage, and without a queueing layer a large team's jobs can starve smaller teams simply by submitting first. A practical policy usually combines hard quotas per team as a ceiling, fair-share weighting for how leftover capacity gets allocated, and a preemption policy for lower-priority jobs so idle-but-reserved capacity does not go to waste. Reporting GPU-hours per team back to stakeholders regularly also reduces disputes, since usage becomes visible rather than a source of friction. Nanobase AI implements fair-share and quota policies tailored to each customer's team structure and workload priorities.

A policy is three separate decisions, not one

Teams often treat "fair scheduling" as a single setting to turn on, but a working policy actually requires three independent decisions layered together: a hard ceiling per team so nobody can consume the entire cluster, a fair-share weighting for how leftover, unreserved capacity gets allocated among active requesters, and a preemption rule for what happens when a high-priority job needs capacity currently held by a lower-priority one. Skipping any one of the three layers is what causes most of the disputes teams associate with "the scheduler isn't fair." A quota alone leaves idle reserved capacity wasted; fair-share alone lets a bursty team starve everyone else during a legitimate but temporary surge; preemption alone without quotas just shifts who complains.

Slurm's fair-share mechanics in practice

Slurm implements this natively through its multifactor priority plugin and association-based fair-share tree. Every account accumulates a usage value from consumed GPU-hours, and Slurm compares that against the account's assigned share to compute a fair-share priority factor that decays over time.

1. Define accounts per team with sacctmgr add account and set a fairshare weight reflecting each team's entitled portion of the cluster.
2. Set PriorityDecayHalfLife in slurm.conf (commonly a few days to a couple of weeks) so historical usage stops penalizing a team once enough time has passed, rather than permanently.
3. Add PriorityWeightFairshare alongside PriorityWeightAge and PriorityWeightJobSize so fair-share is one input among several, not the only factor in job ordering.
4. Set per-account GrpTRES=gres/gpu=N as the hard ceiling that fair-share weighting cannot exceed regardless of priority.
5. Enable PreemptType=preempt/qos with defined QOS tiers so designated high-priority jobs can reclaim capacity from lower-tier jobs cleanly, typically via requeue rather than kill.

Translating the same policy to Kubernetes

Kubernetes has no native concept of historical fair-share; the default scheduler places jobs based on current resource availability, so a team submitting first or in bulk can consume shared GPU capacity regardless of that team's overall entitlement. Reproducing Slurm-like fairness requires an additional layer.

Rolling it out without breaking trust

A policy change to shared infrastructure fails politically more often than it fails technically. Publish the quota and fair-share weights before enabling enforcement, and report actual GPU-hours consumed per team on a recurring cadence so usage is visible rather than a source of after-the-fact disputes. Start preemption in a warn-only or notify mode for a few weeks before enabling automatic requeue, since a team that loses a long-running job to preemption without warning will not trust the policy afterward even if the policy itself is correctly designed. Revisit quota weights quarterly as team headcount and project priority shift, since a policy set once at cluster launch tends to drift out of alignment with actual organizational priorities within a year.

Frequently asked questions

Should quotas be based on GPU count or GPU-hours?

GPU-hours generally produce fairer outcomes than a flat GPU count, since a team running short jobs and a team running long jobs at the same GPU count consume very different amounts of actual capacity. GPU-hour accounting, which both Slurm and Kueue support, rewards efficient short jobs rather than penalizing them for the same quota ceiling as long ones.

What happens to a preempted job?

Well-designed preemption requeues the job rather than killing it outright, meaning it returns to the queue and resumes once capacity frees up, provided the job checkpoints its progress. Preempting a job with no checkpointing wastes the compute already spent, so checkpointing and preemption policy should be designed together.

Do small clusters need a formal fair-share policy?

Clusters under roughly a dozen GPUs shared by two or three teams often function fine with informal coordination and simple hard quotas, since the coordination overhead of a full fair-share tree exceeds its benefit at that scale. Formal fair-share policy earns its complexity once four or more teams compete for capacity regularly.

How Nanobase AI helps

Nanobase AI, a Silicon Valley enterprise AI engineering company, designs quota, fair-share, and preemption policy around each customer's actual team structure and workload mix, then implements it on Slurm, Kueue, or a commercial scheduler depending on scale. That includes the reporting layer that keeps the policy trusted after rollout, not just the initial configuration.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.