Running a single logical Kubernetes GPU cluster that spans on-prem and cloud is possible but usually implemented as a federated or multi-cluster setup rather than one literal control plane, because latency and network reliability between environments make a single stretched control plane fragile. The common approach uses tools such as Cluster API, Karmada, or a service mesh like Istio to unify scheduling and networking across an on-prem cluster running the NVIDIA GPU Operator and a cloud cluster such as EKS, AKS, or GKE with GPU node pools, presenting a consistent API and workload placement policy to users. Amazon's EKS Anywhere and Azure Arc-enabled Kubernetes are built specifically to extend cloud control planes to on-prem GPU hardware, which simplifies identity, monitoring, and policy consistency compared to stitching together fully independent clusters. GPU scheduling features like MIG partitioning and node labeling need to be configured consistently across both environments so workloads land on GPUs with the right memory and isolation guarantees. Network bandwidth and latency between sites remain the main constraint for any workload that needs tight synchronization, such as multi-node training. Nanobase AI builds hybrid Kubernetes GPU clusters using EKS Anywhere, Azure Arc, or comparable federation approaches suited to a customer's existing environment.
Why one stretched control plane is the wrong goal
Teams that ask for one Kubernetes cluster spanning on-premise and cloud are usually really asking for one consistent way to schedule and observe GPU workloads across both environments, not literally one control plane process making decisions across a high-latency, less reliable network link. A single stretched control plane is fragile precisely because Kubernetes' control loop assumes low, consistent latency between components, which a WAN link between a data center and a cloud region rarely provides. The practical answer is federation: multiple independent clusters, each with its own control plane, presenting a unified scheduling and policy interface to users and operators.
Comparing the federation approaches
| Approach | What it provides | Best fit |
|---|---|---|
| EKS Anywhere / Azure Arc-enabled Kubernetes | Extends a cloud control plane's management, policy, and identity to on-prem clusters | Teams already standardized on one hyperscaler wanting consistent tooling |
| Cluster API | Declarative cluster lifecycle management across providers | Teams wanting provider-neutral infrastructure as code |
| Karmada | Multi-cluster application scheduling and propagation | Workloads needing active placement decisions across clusters |
| Service mesh (e.g., Istio multi-cluster) | Unified networking and traffic policy across clusters | Workloads needing cross-cluster service discovery and routing |
No single row in this table replaces the others entirely; most hybrid GPU deployments combine a cluster lifecycle tool with a networking layer, rather than expecting one tool to solve both scheduling and connectivity.
The latency budget that constrains any design
Before selecting a federation approach, the workload's actual latency tolerance needs to be measured, because it determines what kind of cross-cluster interaction is even viable. Stateless inference requests routed to whichever cluster has capacity tolerate the round-trip latency between sites reasonably well, since each request is independent. Multi-node training or inference requiring tight GPU-to-GPU synchronization, by contrast, generally cannot tolerate a WAN hop between GPUs at all and needs to stay within a single cluster's low-latency fabric. Mapping which workloads fall into which latency category before choosing a federation tool avoids designing an elegant multi-cluster architecture that then cannot actually run the workload it was built for.
Configuring GPU scheduling consistently across clusters
A minimal example of labeling GPU nodes consistently across both an on-prem and a cloud cluster, so workload placement rules behave the same way regardless of which cluster they land in, looks like this:
apiVersion: v1
kind: Node
metadata:
labels:
nvidia.com/gpu.product: "NVIDIA-H100-80GB"
topology.kubernetes.io/zone: "on-prem-dc1"Matching labels, MIG partition configuration, and NVIDIA GPU Operator versions across both environments is what allows a single set of scheduling rules, such as node affinity or taints, to behave predictably no matter where a pod actually lands. Skipping this consistency check is a common cause of workloads scheduling successfully but then failing at runtime due to a driver or MIG configuration mismatch between clusters.
Frequently asked questions
Do EKS Anywhere and Azure Arc require the on-prem hardware to be identical to cloud instances?
No, but the NVIDIA GPU Operator version, driver version, and Kubernetes version should be kept consistent across environments to avoid workloads behaving differently depending on which cluster they land in.
Can we federate more than two clusters, such as multiple regions plus on-prem?
Yes, the same federation tools support more than two clusters; the main added complexity is that scheduling and networking logic need to account for more possible placement targets and their respective latency characteristics.
Is MIG configuration required for a hybrid Kubernetes GPU setup?
No, MIG is optional and depends on whether workloads need GPU partitioning for multi-tenancy; when it is used, it needs to be configured consistently across every cluster in the federation to avoid scheduling surprises.
How do we monitor GPU health consistently across on-prem and cloud clusters?
NVIDIA DCGM exporters feeding a centralized Prometheus and Grafana stack, aggregating metrics from every cluster in the federation, is the common approach, giving one dashboard view regardless of where a given GPU physically sits.
How Nanobase AI helps
Nanobase AI builds hybrid Kubernetes GPU clusters using EKS Anywhere, Azure Arc, or comparable federation approaches suited to a customer's existing environment, keeping GPU Operator, driver, and MIG configuration consistent across every cluster in the federation. Our Kubernetes GPU Operator vs Slurm guide covers the scheduling foundation this work builds on.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.