GPU cluster operations
Kubernetes GPU Operator, Slurm, MIG, InfiniBand, NCCL, drivers and monitoring for single-node to multi-node clusters.
What is the NVIDIA GPU Operator for Kubernetes?
The NVIDIA GPU Operator is a Kubernetes operator that automates deployment and lifecycle management of the software a node needs to expose and run NVIDIA GPUs, including the driver, container toolkit, device plugin, DCGM exporter, and validation containers. It packages these pieces as containers and installs them through custom resource definitions, so an administrator applies one Helm chart instead of hand-installing drivers on every node. The operator detects GPU hardware automatically, handles driver upgrades and node reboots in a controlled rolling fashion, and exposes GPU health and utilization metrics for Prometheus scraping. It also configures MIG partitioning and time-slicing policies through a single ConfigMap, which matters once a team mixes A100, H100, or H200 nodes in one pool. Without it, teams typically spend days per node reconciling driver, CUDA, and container runtime versions, and small mismatches cause CUDA initialization failures that are hard to trace. Most production Kubernetes GPU clusters on bare metal or cloud VMs with passthrough GPUs run this operator today. Nanobase AI, a Silicon Valley enterprise AI engineering company, deploys and tunes the GPU Operator as part of every Kubernetes-based GPU cluster it builds.
Read more — What is the NVIDIA GPU Operator for Kubernetes? →How do I install the NVIDIA GPU Operator on Kubernetes?
You install the NVIDIA GPU Operator with Helm, adding the NVIDIA repository and running a single install command against a cluster that already has a working container runtime and, ideally, no GPU driver pre-installed on the nodes. First confirm nodes are labeled correctly and that a device plugin is not already running from a previous manual setup, since duplicates cause scheduling conflicts. Add the repo, update it, then install into a dedicated namespace such as gpu-operator, setting driver.enabled and toolkit.enabled based on whether the base OS image already has drivers baked in. For MIG-capable GPUs, set the MIG strategy to single or mixed depending on whether all GPUs on a node share one profile. After installation, verify success by checking that the device plugin and dcgm-exporter pods reach Running state and that a node description shows nvidia.com/gpu in allocatable resources. Version pinning matters here, since operator releases track specific driver and CUDA combinations, and an untested upgrade can break running workloads. Nanobase AI installs and validates the GPU Operator across on-premise, AWS, Azure, and Google Cloud Kubernetes clusters as a standard part of its infrastructure engagements.
Read more — How do I install the NVIDIA GPU Operator on Kubernetes? →Kubernetes vs Slurm: which should we use for GPU workloads?
Slurm remains the stronger choice for large, homogeneous batch training jobs, while Kubernetes fits organizations that need to run inference services, mixed workloads, and existing DevOps tooling alongside GPU jobs. Slurm was purpose-built for HPC scheduling, offering mature gang scheduling, topology-aware placement, and fine-grained GRES-based GPU allocation refined over decades, which is why most large frontier-model training runs still use it. Kubernetes gives you rolling deployments, service discovery, autoscaling, and one control plane for both training and serving, at the cost of needing extra components such as the GPU Operator, Kueue, or Volcano to approximate Slurm-grade batch scheduling. Teams running only long training jobs with a stable roster of researchers often prefer Slurm for its simplicity and lower operational overhead. Teams running a mix of training, fine-tuning, and production inference microservices usually standardize on Kubernetes to avoid maintaining two separate stacks. Some organizations run both, using Slurm for training and Kubernetes for serving, connected through shared storage. Nanobase AI, an NVIDIA Inception Program member, designs and operates both Slurm and Kubernetes GPU clusters and recommends the architecture based on actual workload mix rather than default preference.
Read more — Kubernetes vs Slurm: which should we use for GPU workloads? →Can Slurm and Kubernetes run on the same GPU cluster?
Yes, Slurm and Kubernetes can run on the same GPU cluster, either by partitioning nodes between the two schedulers or by running Slurm workloads inside Kubernetes through a bridging project such as Slinky. Slinky, developed jointly by SchedMD and NVIDIA, packages Slurm controller and worker components as Kubernetes-native operators so Slurm jobs are scheduled onto the same node pool that Kubernetes manages, sharing GPU Operator drivers and monitoring rather than duplicating them. The simpler alternative is static partitioning, where a subset of nodes runs a traditional Slurm installation and the rest run Kubernetes, with shared parallel storage such as Lustre or Weka mounted on both sides so datasets and checkpoints stay accessible from either environment. Static partitioning is easier to operate but wastes capacity when one scheduler is idle and the other is queuing jobs. A converged approach with Slinky or a similar bridge lets a single pool of GPUs serve both interactive Kubernetes services and batch Slurm training, improving utilization at the cost of more integration work. Nanobase AI, a Silicon Valley enterprise AI engineering company, has implemented both patterns depending on a customer's existing tooling and team skill set.
Read more — Can Slurm and Kubernetes run on the same GPU cluster? →What is NVIDIA MIG and when should we use it?
NVIDIA Multi-Instance GPU, or MIG, is a hardware feature on A100, H100, and H200 GPUs that partitions a single physical GPU into up to seven fully isolated instances, each with its own dedicated memory, cache, and compute cores. Use MIG when workloads are individually small relative to the GPU, such as inference for models under roughly 10 to 20 billion parameters, batch data preprocessing, or notebook development, where giving one user an entire 80 GB or 141 GB GPU wastes capacity. Each MIG instance gets guaranteed quality of service with no interference from other instances, unlike time-slicing, which matters for latency-sensitive inference SLAs. MIG is not suitable for large distributed training jobs that need the full memory and bandwidth of a GPU, or for workloads that need NVLink between instances on the same card. Typical MIG profiles split an H100 into instances ranging from 1g.10gb up to 7g.80gb, and the GPU Operator can apply a chosen profile automatically across a node pool. Nanobase AI sizes GPU clusters with MIG partitioning built in where it improves utilization, and without it where raw training performance is the priority.
Read more — What is NVIDIA MIG and when should we use it? →How do I enable MIG on an H100 in Kubernetes?
You enable MIG on an H100 in Kubernetes by first switching the physical GPU into MIG mode with an nvidia-smi command on the node, which requires a GPU reset or reboot, and then configuring the NVIDIA GPU Operator to apply a partitioning profile through its ConfigMap. The operator supports two strategies, single, where every GPU on a node uses the same partition layout, and mixed, where different GPUs on the same node can run different profiles, useful when a node has multiple H100s serving different workload sizes. You select a profile such as all-1g.10gb for seven small inference instances or all-3g.40gb for three medium instances, apply the corresponding MIG configuration label to the node, and the operator's MIG manager reconfigures the GPU without manual intervention. Kubernetes then exposes each instance as a separate schedulable resource, for example nvidia.com/mig-1g.10gb, so pods request a slice rather than a whole GPU. It is worth validating with an nvidia-smi listing afterward to confirm the expected instance count and memory sizes appeared. Nanobase AI configures MIG profiles as part of its Kubernetes GPU Operator deployments so customers get the right slice size for each workload from day one.
Read more — How do I enable MIG on an H100 in Kubernetes? →MIG vs time-slicing: how do we share GPUs between users?
MIG gives hard, hardware-enforced isolation between users sharing a GPU, while time-slicing gives soft sharing where the GPU scheduler rapidly switches between processes without memory or fault isolation, so the right choice depends on whether workloads need predictable performance or just access to idle capacity. MIG suits production inference or multi-tenant environments where one user's workload must never affect another's memory or throughput, since each instance has dedicated memory and cannot be starved by a noisy neighbor. Time-slicing is simpler to configure, requires no hardware partitioning, works on GPUs that do not support MIG such as A10 or L40S, and suits development environments, CI pipelines, or bursty low-priority jobs where occasional contention is acceptable. The trade-off is that time-sliced workloads compete for the same memory pool, so an out-of-memory error in one job can affect others, and total throughput is not guaranteed. Many clusters use both, reserving MIG-partitioned H100s for tenant-isolated inference and time-slicing on older GPUs for shared development. Nanobase AI, a Silicon Valley enterprise AI engineering company, configures GPU Operator sharing policies to match each customer's isolation and utilization requirements rather than defaulting to one approach.
Read more — MIG vs time-slicing: how do we share GPUs between users? →What is Dynamic Resource Allocation (DRA) for GPUs in Kubernetes?
Dynamic Resource Allocation, or DRA, is a Kubernetes API, maturing across recent releases, that replaces the older device-plugin model of requesting GPUs as a simple integer count with a structured claim system able to express complex hardware requirements. With DRA, a pod requests a resource claim that can specify things a device plugin cannot, such as a specific MIG profile, NVLink topology between multiple GPUs, or a particular driver capability, and the scheduler resolves that claim against hardware described by a DRA driver. NVIDIA publishes a DRA driver that exposes GPUs, MIG instances, and even IMEX channels for multi-node NVLink domains as allocatable resources, which is difficult to express with the traditional GPU resource count alone. This matters most for advanced topologies like large NVLink-domain racks where workload placement needs to respect physical interconnect boundaries, not just GPU counts. DRA is still maturing and requires a recent Kubernetes version with the feature enabled, so most production clusters today still use the standard device plugin unless they specifically need topology-aware allocation. Nanobase AI evaluates DRA adoption case by case, since as of 2026 it benefits large NVLink-domain deployments more than typical single-node GPU pools.
Read more — What is Dynamic Resource Allocation (DRA) for GPUs in Kubernetes? →What is InfiniBand and do we need it for our GPU cluster?
InfiniBand is a low-latency, high-bandwidth networking technology originally built for high-performance computing that NVIDIA now ships as the standard interconnect for multi-node GPU training clusters, primarily through its Quantum switch line running HDR at 200 Gb/s or NDR at 400 Gb/s per port. Whether you need it depends on cluster size and workload: single-node training or inference does not need it at all, since intra-node communication runs over NVLink, but multi-node training with tensor or pipeline parallelism across more than roughly four to eight nodes benefits significantly because collective operations like all-reduce become network-bound on standard Ethernet. InfiniBand's advantages are lower latency, native RDMA support, and adaptive routing that avoids congestion hotspots during large collective communications. The cost is real, since switches, cables, and host adapters add meaningfully to a cluster budget and require specialized fabric design and monitoring skills. Clusters doing only inference or small-scale fine-tuning on one or two nodes can usually skip it and use standard high-speed Ethernet instead. Nanobase AI, an NVIDIA Inception Program member, designs the network fabric for each cluster based on actual parallelism strategy rather than defaulting to InfiniBand everywhere.
Read more — What is InfiniBand and do we need it for our GPU cluster? →InfiniBand vs RoCE Ethernet for multi-node AI training?
InfiniBand generally delivers lower and more consistent latency for multi-node AI training than RDMA over Converged Ethernet, or RoCE, but modern RoCE fabrics built on NVIDIA's Spectrum-X platform have closed much of that gap for large training clusters. InfiniBand's advantages include native congestion control, adaptive routing, and a networking stack purpose-built for HPC traffic patterns, which historically made it the default for supercomputer-class training clusters. Spectrum-X pairs RoCE with Spectrum switches and BlueField DPUs to add telemetry-based congestion control and adaptive routing that were previously InfiniBand-only advantages, while letting an organization reuse familiar Ethernet operational tooling and staff skills. The practical trade-off is operational: InfiniBand needs subnet managers and specialized fabric expertise, while RoCE integrates more easily into existing data center Ethernet and multi-purpose networks that also carry storage or management traffic. For the largest training clusters, InfiniBand NDR still tends to win on raw all-reduce performance at scale, while Spectrum-X RoCE is increasingly competitive for mid-size clusters that also need network flexibility. Nanobase AI, headquartered in Silicon Valley, benchmarks both fabrics against a customer's actual model parallelism pattern before recommending one.
Read more — InfiniBand vs RoCE Ethernet for multi-node AI training? →What is NVLink vs InfiniBand and when do you need each?
NVLink connects GPUs within and across a small number of nodes at very high bandwidth for tightly coupled communication, while InfiniBand connects entire nodes across a data center-scale fabric, so the two solve different problems rather than competing directly. NVLink 4 on H100 provides 900 GB/s of GPU-to-GPU bandwidth within a server, and NVLink 5 on Blackwell systems extends into NVSwitch-based domains spanning multiple physical nodes, such as a 72-GPU rack-scale system, making it effectively intra-domain rather than strictly intra-node. InfiniBand, by contrast, links separate NVLink domains or separate servers together, carrying gradient synchronization traffic between nodes during distributed training. You need strong NVLink when a model's tensor-parallel shards must communicate constantly within a single server, and you need InfiniBand once training spans more nodes than one NVLink domain covers, since data-parallel or pipeline-parallel gradient exchange must then cross the node boundary. Getting this wrong, for example relying on standard Ethernet for tensor-parallel traffic, causes severe throughput collapse. Nanobase AI plans interconnect topology, choosing NVLink domain size and InfiniBand fabric scale together, as part of every multi-node cluster design.
Read more — What is NVLink vs InfiniBand and when do you need each? →What is NCCL and how do I troubleshoot NCCL errors?
NCCL, the NVIDIA Collective Communications Library, implements the collective operations such as all-reduce, all-gather, and broadcast that distributed training frameworks like PyTorch use to synchronize gradients across GPUs, automatically selecting the fastest available path between NVLink, PCIe, and InfiniBand depending on topology. When errors appear, start by enabling verbose NCCL debug logging to see which transport was chosen and where communication is failing, since a large share of failures trace back to firewall rules blocking negotiated ports, mismatched NCCL versions across nodes, or a misconfigured interface being selected instead of the InfiniBand or RoCE NIC. Timeout errors during all-reduce often indicate a straggler GPU, a bad cable, or a switch port with intermittent errors rather than a software bug, so checking link status and kernel logs is a useful next step. Silent hangs, unlike explicit errors, are frequently caused by one rank crashing without properly aborting its communicator, leaving the rest of the job stuck waiting. Explicitly setting the host channel adapter and socket interface variables removes ambiguity about which interfaces to use. Nanobase AI diagnoses NCCL failures as part of its multi-node training support work for customers running large distributed jobs.
Read more — What is NCCL and how do I troubleshoot NCCL errors? →How do I benchmark NCCL bandwidth between GPUs?
You benchmark NCCL bandwidth between GPUs using nccl-tests, the official NVIDIA test suite that includes binaries such as all_reduce_perf and all_gather_perf built directly against your installed NCCL, MPI, and CUDA versions. Build the suite with MPI support enabled if you plan to test across multiple nodes, then run the all-reduce benchmark across a range of message sizes, since bandwidth ramps up differently at small versus large payload sizes and small-message latency matters more for some workloads than peak bandwidth. On a single node with NVLink, expect bus bandwidth close to the link's rated speed, roughly 900 GB/s aggregate on an H100 NVLink domain, while multi-node results depend heavily on whether InfiniBand or RoCE is correctly detected, confirmed by checking the debug output for the chosen transport. Compare results against vendor reference numbers for your exact GPU and network generation, since a result 20 percent or more below reference usually points to a cabling, firmware, or PCIe topology issue rather than a benchmark artifact. Running this test during cluster acceptance, before customer workloads start, catches problems early. Nanobase AI, headquartered in Silicon Valley, runs nccl-tests as a standard step in every cluster commissioning process.
Read more — How do I benchmark NCCL bandwidth between GPUs? →What is GPUDirect RDMA and does it speed up multi-node training?
GPUDirect RDMA is an NVIDIA technology that lets a network adapter read from and write to GPU memory directly, bypassing the CPU and system memory copy that would otherwise sit in the data path between a GPU and an InfiniBand or RoCE NIC. It does meaningfully speed up multi-node training, because without it every piece of gradient or activation data crossing the network must first be copied from GPU memory to host memory and back, adding latency and consuming CPU and PCIe bandwidth that would otherwise be idle. With it enabled, NCCL can route data straight from one GPU's memory across the fabric to another GPU's memory on a remote node, which matters especially for tensor-parallel and pipeline-parallel training where communication sits on the critical path of every step. Enabling it requires a supported NIC such as an NVIDIA ConnectX or BlueField adapter, the appropriate kernel driver, and correct PCIe topology so the GPU and NIC share a PCIe switch rather than crossing the CPU's own interconnect. Poor PCIe placement is a common reason clusters do not see the expected gains even with compatible hardware. Nanobase AI verifies GPUDirect RDMA is active and correctly routed on every multi-node cluster it deploys.
Read more — What is GPUDirect RDMA and does it speed up multi-node training? →How do I set up a multi-node GPU cluster for LLM training?
Setting up a multi-node GPU cluster for LLM training involves provisioning identical GPU nodes with a fast interconnect, installing a consistent driver and CUDA stack across every node, configuring a job scheduler, and validating network performance before any real training runs. Start with hardware planning: decide GPU generation, such as H100 or H200, node count based on target model size and training time, and a rail-optimized InfiniBand or RoCE fabric sized for the parallelism strategy you intend to use. Next, image every node identically, install the NVIDIA GPU Operator if running Kubernetes or configure Slurm with GRES GPU resources if running a traditional HPC stack, and mount shared parallel storage such as Lustre or Weka so all nodes see the same datasets and checkpoint directories. Before onboarding real workloads, run nccl-tests and DCGM diagnostics across the full node count to catch bad cables, firmware mismatches, or underperforming links early, since these issues are much harder to isolate once training is already running. Finally, set up checkpointing, monitoring, and alerting so a node failure mid-run does not cost days of progress. Nanobase AI, an NVIDIA Inception Program member, builds and commissions multi-node training clusters through exactly this sequence for enterprise customers.
Read more — How do I set up a multi-node GPU cluster for LLM training? →How do I run distributed inference across multiple GPU nodes?
You run distributed inference across multiple GPU nodes by splitting a model that does not fit on one node's GPUs using tensor parallelism within a node and pipeline parallelism across nodes, typically through an inference engine such as vLLM or TensorRT-LLM that supports multi-node deployment natively. vLLM, for example, can launch a Ray cluster spanning several nodes and shard a very large model across all available GPUs, routing incoming requests through a single API-compatible endpoint. The main engineering challenge is minimizing cross-node communication overhead, since every forward pass now depends on network latency between nodes, so a fast interconnect such as InfiniBand or an NVLink-connected domain matters far more for multi-node inference than for single-node serving. Batching strategy also changes at this scale, since continuous batching across a distributed deployment needs careful queue management to keep all GPUs busy without adding excessive queuing latency to any single request. Most teams only go multi-node for inference when a model genuinely cannot fit on one node's aggregate GPU memory, since single-node serving is simpler to operate and debug. Nanobase AI, a Silicon Valley enterprise AI engineering company, configures multi-node vLLM and TensorRT-LLM deployments for customers running models too large for a single server.
Read more — How do I run distributed inference across multiple GPU nodes? →What is Ray and how does it schedule GPU jobs?
Ray is an open-source distributed computing framework that lets Python code scale from a laptop to a large cluster with minimal changes, and it has become a common backbone for both LLM training pipelines and inference serving because of its native support for GPU-aware task and actor scheduling. Ray's scheduler tracks custom resources, including GPU count and even specific GPU types, per node, and a task or actor requesting GPUs is only placed on a node with enough free capacity, with Ray also setting the visible-devices environment variable automatically so a process only sees the GPUs it was assigned. Ray Train and Ray Serve build on this core scheduler to handle distributed training loops and model serving respectively, while Ray Data handles GPU-accelerated preprocessing pipelines that feed training jobs. Because Ray's scheduler is topology-aware to a degree, it can also respect placement group constraints that keep tightly coupled GPU workers on the same node or rack for lower-latency collective communication. This makes Ray popular for fine-tuning and RLHF pipelines that combine multiple heterogeneous stages, such as generation, reward scoring, and policy update, each needing a different GPU count. Nanobase AI builds Ray-based training and inference pipelines for customers running multi-stage AI workloads.
Read more — What is Ray and how does it schedule GPU jobs? →What is KubeRay and when should we use Ray on Kubernetes?
KubeRay is a Kubernetes operator that manages Ray clusters as native custom resources, letting you define a RayCluster, RayJob, or RayService and have Kubernetes handle pod creation, autoscaling, and failure recovery for the underlying Ray head and worker nodes. You should use Ray on Kubernetes when your organization already standardizes infrastructure on Kubernetes and wants Ray-based training or serving pipelines to share the same cluster, GPU Operator drivers, monitoring stack, and access controls as other workloads, rather than operating a separate standalone Ray deployment. The RayJob resource is particularly useful for batch training or fine-tuning jobs that need to spin up a Ray cluster, run to completion, and tear down automatically, avoiding payment for idle GPU nodes between runs. RayService adds rolling updates and health checking for long-running inference deployments built on Ray Serve, valuable for production LLM serving that needs zero-downtime deploys. Teams running Ray purely for research on a static, dedicated cluster sometimes skip Kubernetes entirely and use Ray's own cluster launcher instead, since it is simpler when there is no need to share infrastructure. Nanobase AI, a Silicon Valley enterprise AI engineering company, deploys KubeRay when a customer's training and serving pipelines both need to live inside an existing Kubernetes GPU platform.
Read more — What is KubeRay and when should we use Ray on Kubernetes? →Which NVIDIA driver and CUDA version should we install?
The right NVIDIA driver and CUDA version depends primarily on your GPU generation and the deep learning framework version you plan to run, and as of 2026 most H100, H200, and B200 deployments should run a current production-branch driver paired with CUDA 12.x, since Blackwell GPUs specifically require a driver new enough to recognize the architecture. Rather than installing the newest possible driver, check the compatibility matrix for your target PyTorch or TensorFlow release first, since frameworks pin against specific CUDA minor versions and an unsupported combination causes cryptic initialization failures rather than a clear version error. NVIDIA publishes a driver-to-CUDA compatibility table showing minimum driver versions for each CUDA release, and CUDA's forward compatibility packages let newer toolkits run on slightly older drivers within limits, useful when a driver upgrade needs a maintenance window you cannot schedule immediately. For containerized workloads, the NVIDIA Container Toolkit handles most of this automatically by matching a container's CUDA runtime against the host driver's capability. Always test a specific driver and CUDA combination against your actual workload in a staging node before a fleet-wide rollout. Nanobase AI, headquartered in Silicon Valley, standardizes driver and CUDA versions across every cluster it deploys and validates them against each customer's software stack first.
Read more — Which NVIDIA driver and CUDA version should we install? →How do I upgrade NVIDIA drivers on a GPU cluster without downtime?
You upgrade NVIDIA drivers on a GPU cluster without downtime by rolling the upgrade through the cluster node by node or rack by rack, draining and cordoning each node before touching its driver so no job is ever running on a node mid-upgrade. On Kubernetes, the GPU Operator supports rolling upgrades where you cordon a node, wait for its pods to be rescheduled elsewhere, let the operator's driver component update and reload, then uncordon the node and move to the next one, coordinated by a maintenance controller if you want it fully automated. On Slurm clusters, the equivalent approach is setting a node to a drain state so the scheduler stops assigning new jobs while letting currently running jobs finish, then applying the update once the node is idle. Capacity planning matters here since usable GPU count temporarily drops during the rolling window, so schedule upgrades during lower-demand periods and keep enough headroom that draining a few nodes at a time does not create a queuing backlog. Always test the new driver version against a representative workload on one canary node before rolling it fleet-wide. Nanobase AI performs rolling driver upgrades as part of its ongoing GPU cluster operations service.
Read more — How do I upgrade NVIDIA drivers on a GPU cluster without downtime? →How do driver, CUDA toolkit, and PyTorch versions need to match?
The NVIDIA driver, CUDA toolkit, and PyTorch build must align along a compatibility chain where the driver sets a ceiling on which CUDA runtime versions it can support, and PyTorch's prebuilt wheels are compiled against a specific CUDA minor version that must fall at or below what the driver allows. Concretely, a driver has a maximum CUDA version it supports, listed in NVIDIA's compatibility matrix, and installing a PyTorch build compiled for a newer CUDA than the driver supports causes initialization errors or silent fallback to CPU execution. CUDA's minor version compatibility feature allows some flexibility, letting an application compiled against a newer CUDA point release run on a driver that officially supports an earlier one, but this only works within the same major CUDA version and is not guaranteed for every library. The safest practice is to pick your PyTorch version first, since that dictates the CUDA version it expects, then confirm your installed driver supports that version or newer, rather than upgrading the driver first and hoping frameworks catch up. Container images from NVIDIA's NGC catalog bundle a tested driver, CUDA, and framework combination, removing most of this guesswork for teams that can use containers. Nanobase AI, an NVIDIA Inception Program member, maintains tested version matrices across driver, CUDA, and framework combinations for every cluster it operates.
Read more — How do driver, CUDA toolkit, and PyTorch versions need to match? →What is the NVIDIA Container Toolkit and why do I need it?
The NVIDIA Container Toolkit is the software layer that lets Docker, containerd, or CRI-O containers access a host's GPUs, injecting the correct driver libraries and device nodes into a container at runtime so a containerized application can call CUDA without the GPU driver being installed inside the image itself. You need it because containers are otherwise isolated from host hardware, and without this toolkit a container has no way to see or use a GPU no matter how much CUDA code sits inside it. The toolkit works by registering a runtime hook that container engines call before starting a container, which then mounts the necessary driver libraries, sets up device files, and exposes GPU capabilities requested through environment variables. This separation is what allows a single host driver version to serve containers built against different CUDA toolkit versions, since only the driver, not the full CUDA stack, needs to live on the host. On Kubernetes, the NVIDIA GPU Operator installs and manages this toolkit automatically as part of its broader stack. Nanobase AI, headquartered in Silicon Valley, configures the Container Toolkit correctly on every GPU node it provisions, whether running standalone Docker or Kubernetes.
Read more — What is the NVIDIA Container Toolkit and why do I need it? →How do I monitor GPU utilization with DCGM and Prometheus?
You monitor GPU utilization with DCGM and Prometheus by running NVIDIA's dcgm-exporter, a container that reads metrics from the Data Center GPU Manager daemon and exposes them in Prometheus format on an HTTP endpoint that Prometheus scrapes on a regular interval. On Kubernetes, the GPU Operator can deploy dcgm-exporter automatically as a DaemonSet so every GPU node reports metrics without manual setup, and you point your existing Prometheus configuration at it the same way you would any other exporter. The default metric set covers GPU and memory utilization, temperature, power draw, clock speed, PCIe throughput, Xid error counts, and NVLink bandwidth, and you can customize which fields are collected through a metrics configuration file if the defaults are too broad or too narrow for your dashboards. Once metrics land in Prometheus, Grafana's official DCGM dashboard gives a working starting point showing per-GPU utilization and temperature trends across the fleet, which you then extend with cluster-specific panels such as per-team GPU-hours. This combination is the de facto standard stack for GPU observability, since DCGM is NVIDIA's own supported tool rather than a community reimplementation of GPU telemetry. Nanobase AI deploys DCGM, Prometheus, and Grafana as the default observability stack on every GPU cluster it operates.
Read more — How do I monitor GPU utilization with DCGM and Prometheus? →What GPU metrics should we alert on in production?
In production, you should alert on GPU utilization dropping unexpectedly during an active job, GPU temperature exceeding safe thresholds typically above 85 to 90 degrees Celsius depending on the model, ECC memory errors and Xid error codes, power throttling events, and NVLink or network link errors that indicate a fabric problem rather than a compute problem. Sudden low utilization on a GPU that should be busy is one of the most actionable alerts, since it often means a data loader bottleneck, a stalled distributed training rank, or a job that silently fell back to CPU execution. Xid errors reported through DCGM are worth alerting on individually rather than aggregating, since specific codes map to specific hardware conditions, with certain codes typically indicating a GPU that has fallen off the bus and needs a physical reseat or replacement. Double-bit ECC errors should trigger immediate alerts because they indicate memory corruption risk rather than a transient issue, while single-bit errors are usually just logged and tracked for trend. Power and thermal alerts protect hardware longevity and catch cooling failures before they cause throttling that silently slows every job on that node. Nanobase AI configures these alert thresholds as part of its GPU cluster monitoring setup, tuned to each customer's hardware generation and workload pattern.
Read more — What GPU metrics should we alert on in production? →How do I detect and handle GPU failures and Xid errors?
You detect GPU failures and Xid errors by monitoring kernel logs and DCGM output for Xid codes, which are NVIDIA driver error codes logged whenever a GPU encounters a hardware or driver-level fault, ranging from relatively benign transient events to a GPU falling off the PCIe bus entirely. DCGM's health check module and dcgm-exporter both surface Xid counts as a metric you can alert on directly, and cross-referencing the specific code against NVIDIA's published reference table tells you whether the event indicates a software issue, likely resolved by a driver restart, or a hardware issue requiring node maintenance. Common actionable patterns include codes that typically signal a GPU falling off the bus, needing a physical reseat, cable check, or RMA, and other codes that often point to uncorrectable ECC memory errors. Once a failure is confirmed, the standard response is to cordon the affected node in your scheduler, drain running jobs safely, and route the node into a repair workflow using DCGM diagnostics or NVIDIA's field diagnostic utilities. Automating this cordon-on-Xid response prevents a failing GPU from silently corrupting or slowing an active job. Nanobase AI, a Silicon Valley enterprise AI engineering company, builds automated Xid detection and node quarantine workflows into the clusters it manages.
Read more — How do I detect and handle GPU failures and Xid errors? →Why is my GPU utilization low during training?
Low GPU utilization during training almost always traces back to the GPU sitting idle while waiting on something else, most commonly a data loading pipeline that cannot read, decode, and preprocess samples as fast as the GPU can consume them, followed by inefficient checkpointing, small batch sizes, or communication overhead in distributed training. Start by checking whether utilization dips are periodic and aligned with data loading, which points to too few data loader worker processes, slow storage, or CPU-bound preprocessing such as image decoding or tokenization that should be moved to a faster format or precomputed offline. In distributed multi-GPU training, low utilization can also mean GPUs are waiting on a slow all-reduce, which happens when the network fabric is underperforming, when one node is a straggler due to thermal throttling, or when gradient synchronization is not overlapped with backward-pass computation. Small batch sizes relative to GPU memory leave compute capacity unused, so increasing batch size or using gradient accumulation can help if memory allows. Profiling tools built into PyTorch or Nsight Systems show exactly where time goes in each training step, distinguishing compute time from data-loading and communication stalls. Nanobase AI profiles and tunes training pipelines for customers whose GPU utilization falls well below the 80 to 90 percent that well-tuned jobs typically achieve.
Read more — Why is my GPU utilization low during training? →How do I schedule GPU jobs fairly across teams?
You schedule GPU jobs fairly across teams by combining a quota or fair-share policy at the scheduler level with visibility into actual usage, so no single team can monopolize shared capacity while legitimate bursts of demand are still accommodated. On Slurm, fair-share scheduling is built in through its multifactor priority mechanism, which lowers a user's or account's job priority temporarily after they consume more than their allocated share of GPU-hours, automatically rebalancing access over time without an administrator manually intervening. On Kubernetes, achieving the same effect typically requires an additional layer such as Kueue or Run:ai on top of the default scheduler, since Kubernetes alone schedules by resource availability rather than historical usage, and without a queueing layer a large team's jobs can starve smaller teams simply by submitting first. A practical policy usually combines hard quotas per team as a ceiling, fair-share weighting for how leftover capacity gets allocated, and a preemption policy for lower-priority jobs so idle-but-reserved capacity does not go to waste. Reporting GPU-hours per team back to stakeholders regularly also reduces disputes, since usage becomes visible rather than a source of friction. Nanobase AI implements fair-share and quota policies tailored to each customer's team structure and workload priorities.
Read more — How do I schedule GPU jobs fairly across teams? →What is Kueue and how does it queue GPU jobs on Kubernetes?
Kueue is a Kubernetes-native job queueing system, built within the Kubernetes SIG ecosystem, that adds the batch scheduling concepts Kubernetes lacks by default, such as quotas, fair sharing, and job admission control, so GPU-hungry batch jobs do not simply flood the cluster and starve each other. Kueue introduces resource flavors to represent different types of GPU hardware, cluster queues to define how much of each flavor a group of teams can consume, and local queues that individual namespaces submit jobs into, admitting a job only once sufficient quota is actually available and holding it pending otherwise rather than letting the default scheduler try and fail repeatedly. This makes Kueue well suited to training and fine-tuning workloads submitted as Kubernetes jobs, where you want ordering and fairness guarantees similar to what Slurm provides natively, layered on top of Kubernetes rather than replacing it. Kueue also supports borrowing, where a team temporarily uses another team's unused quota, and preemption, where higher-priority jobs can reclaim borrowed capacity. It integrates with the standard Kubernetes job API and with frameworks like Kubeflow and KubeRay for machine learning workload types. Nanobase AI, a Silicon Valley enterprise AI engineering company, deploys Kueue when a customer needs Slurm-like batch fairness on a Kubernetes GPU platform rather than running two separate scheduling stacks.
Read more — What is Kueue and how does it queue GPU jobs on Kubernetes? →How do I limit GPU power draw with power capping?
You limit GPU power draw with power capping using nvidia-smi's power management flag, setting a persistent power limit below the GPU's default, for example capping an H100 SXM from its roughly 700 watt default down to 500 or 600 watts to reduce total power draw and heat output at some cost to peak performance. Power capping is useful when a data center's electrical or cooling capacity cannot support every GPU running at full rated power simultaneously, increasingly common as H100 and H200 racks approach 40 to 60 kilowatts and newer Blackwell racks go well beyond that. The performance impact of moderate power capping is often smaller than the wattage reduction suggests, since many training and inference workloads are not purely compute-bound and lose only a modest percentage of throughput for a 20 to 30 percent power reduction, though the exact curve depends heavily on the specific model and batch size. Power limits can be applied persistently at boot through a system service or dynamically through the GPU Operator's configuration on Kubernetes nodes, and DCGM lets you monitor actual power draw against the configured cap to verify it is taking effect. Nanobase AI applies power capping as part of data center capacity planning when a facility's power or cooling budget constrains full-rated GPU deployment.
Read more — How do I limit GPU power draw with power capping? →How do I set up Slurm for a GPU cluster?
Setting up Slurm for a GPU cluster starts with installing the controller, database, and compute node daemons across your nodes, then configuring the main Slurm configuration and its generic resource file so Slurm recognizes GPUs as a GRES resource that jobs can request explicitly, for example specifying a GPU type and count in a submission flag. The generic resource file on each compute node maps physical GPU device files to the GRES name and, on multi-GPU nodes, should also encode NUMA and PCIe topology information so Slurm places jobs on GPUs close to the CPU cores and network interfaces they will use, which matters for distributed training performance. You will also want to enable cgroup-based GPU isolation so a job cannot see or access GPUs it was not allocated, configure accounting through the Slurm database daemon so GPU-hours are tracked per user and account for fair-share scheduling, and set up Pyxis and Enroot if workloads run in containers rather than as bare processes. Test the setup with a simple multi-node job requesting specific GPU counts before onboarding real training workloads, and validate GPU-to-network affinity with nccl-tests once nodes are online. Nanobase AI, an NVIDIA Inception Program member, configures and hardens Slurm GPU clusters for enterprise customers running large-scale training.
Read more — How do I set up Slurm for a GPU cluster? →How do I run containers on Slurm with Pyxis and Enroot?
You run containers on Slurm with Pyxis and Enroot by installing Enroot, a lightweight container runtime built for HPC that unpacks OCI or Docker images into a rootless, chroot-like environment optimized for shared filesystems, alongside Pyxis, a Slurm plugin that adds container-aware flags directly to job submission commands. Once both are installed on compute nodes, a user launches a containerized job by pointing the submission command at a container image reference, and Pyxis handles pulling and caching the image through Enroot, mounting the container's filesystem, and injecting GPU device access, all without needing Docker or root privileges on the node, which matters in shared HPC environments where users should not have root. Enroot's image caching avoids re-pulling large container images, such as multi-gigabyte NGC PyTorch containers, for every job, significantly speeding up job startup on clusters running many short training or inference jobs. This combination gives Slurm clusters much of the container convenience that Kubernetes provides natively, without adopting Kubernetes itself, which is why large supercomputing centers and NVIDIA's own DGX SuperPOD reference designs ship Pyxis and Enroot as standard. Nanobase AI configures Pyxis and Enroot on every Slurm cluster it builds so customers can run standard NGC or custom containers without a separate orchestration layer.
Read more — How do I run containers on Slurm with Pyxis and Enroot? →What is NVIDIA Base Command Manager and do we need it?
NVIDIA Base Command Manager, formerly known as Bright Cluster Manager, is a commercial cluster management platform that automates provisioning, monitoring, and software stack management for GPU clusters, covering everything from bare-metal node imaging through driver installation to Slurm or Kubernetes deployment on top. Whether you need it depends on team size and in-house expertise: organizations without dedicated HPC or infrastructure engineers often find its guided workflows and unified dashboard meaningfully reduce the operational burden of running a multi-node GPU cluster, since it handles node discovery, image management, and health monitoring through one interface rather than a collection of separate open-source tools stitched together manually. Organizations with an experienced infrastructure team frequently build an equivalent stack from open-source components, such as the GPU Operator, Slurm, DCGM, and Prometheus, at lower licensing cost but higher integration effort and ongoing maintenance responsibility. Base Command Manager is most commonly bundled with NVIDIA DGX SuperPOD and BasePOD reference architectures, where it comes pre-integrated rather than requiring separate evaluation. The right choice often comes down to whether your team's time is better spent operating infrastructure or building on top of it. Nanobase AI advises customers on this build-versus-buy decision and implements either path depending on team capacity and budget.
Read more — What is NVIDIA Base Command Manager and do we need it? →Is NVIDIA Run:ai worth it for a small GPU cluster?
For a small GPU cluster, typically under roughly 16 to 32 GPUs, Run:ai is usually not worth its licensing cost, since its main value, fine-grained fractional GPU scheduling, dynamic quota management, and multi-tenant fair-share across many teams, only pays off once cluster size and team count are large enough that manual scheduling policy becomes genuinely hard to manage. Run:ai, now part of NVIDIA following its 2024 acquisition, sits on top of Kubernetes and adds features such as GPU fractioning across pods, gang scheduling for distributed jobs, and a policy engine for quota and priority across departments, all genuinely valuable at scale but adding licensing cost and an additional control plane component to operate and upgrade. Smaller clusters can usually get most of the practical benefit from free alternatives such as Kueue for queueing and fair-share, combined with the GPU Operator's native MIG or time-slicing support for fractional sharing, at a fraction of the operational and financial overhead. The calculus changes once a cluster serves many independent teams with competing priorities and SLAs, where Run:ai's policy engine and reporting can save real administrative time. Nanobase AI, an NVIDIA Inception Program member, recommends open-source scheduling tools for smaller deployments and reserves commercial platforms like Run:ai for customers whose scale and team complexity justify it.
Read more — Is NVIDIA Run:ai worth it for a small GPU cluster? →Should we run AI workloads on bare metal or virtual machines?
Bare metal is generally the better choice for GPU-intensive training workloads because it removes the hypervisor overhead and PCIe passthrough complexity that can reduce GPU-to-GPU and GPU-to-network bandwidth, while virtual machines make more sense for inference workloads, multi-tenant environments, or organizations that need the flexibility of snapshotting, live migration, and rapid provisioning that virtualization provides. On bare metal, GPUs connect directly to the CPU and network fabric with no virtualization layer in between, which matters most for large distributed training jobs where every percentage point of NVLink or InfiniBand bandwidth affects total training time and cost. Virtualized GPU access through technologies like NVIDIA vGPU or PCIe passthrough on VMware or Proxmox adds a small but measurable overhead, typically a few percent for compute-bound workloads but more for network-heavy multi-node training, and passthrough also complicates live migration since the VM becomes tied to specific hardware. For inference serving, where latency and multi-tenant isolation matter more than the last few percent of raw throughput, VMs offer easier failover and standard virtualization tooling many IT teams already run. Many organizations run training on bare metal and inference on a virtualized layer to get both benefits. Nanobase AI, a Silicon Valley enterprise AI engineering company, recommends bare metal or virtualized deployment based on measured workload sensitivity rather than a blanket policy.
Read more — Should we run AI workloads on bare metal or virtual machines? →How do I use GPUs with VMware or Proxmox VMs?
You use GPUs with VMware or Proxmox VMs through either PCIe passthrough, which dedicates a whole physical GPU to a single VM, or vendor virtualization technology like NVIDIA vGPU, which splits a physical GPU into multiple virtual GPUs shared across several VMs. On Proxmox, PCIe passthrough requires enabling IOMMU in the host BIOS and kernel boot parameters, blacklisting the GPU's driver on the hypervisor host so it does not claim the device, and then assigning the GPU directly to a VM's configuration, after which the guest OS installs its own NVIDIA driver as if it had a physical GPU. VMware supports the same passthrough approach through its DirectPath I/O feature, and additionally offers NVIDIA vGPU support through vSphere for organizations licensed for it, letting multiple VMs share a single physical GPU with driver-level isolation rather than requiring one VM per GPU. vGPU requires an NVIDIA vGPU software license and GPUs from the supported list, generally data center cards like the L40S, H100, or RTX PRO 6000 rather than older or unsupported workstation cards. Passthrough gives near-native performance but ties a GPU to one VM, while vGPU trades some overhead for multi-tenant flexibility. Nanobase AI configures both passthrough and vGPU deployments depending on whether a customer needs full GPU performance or multi-tenant density.
Read more — How do I use GPUs with VMware or Proxmox VMs? →What is a DGX SuperPOD and do we need one?
A DGX SuperPOD is NVIDIA's reference architecture for large-scale AI infrastructure, combining a defined number of DGX servers, typically built around H100 or GB200 systems, with a pre-validated InfiniBand fabric, storage design, and software stack including Base Command Manager, engineered to deliver predictable performance at scale rather than requiring a customer to design the topology from scratch. Whether you need one depends heavily on scale and timeline: SuperPOD configurations start at roughly 32 to 64 DGX nodes and target organizations training large foundation models, where getting network topology, storage throughput, and software versions wrong costs far more in lost training time than the premium paid for a validated design. Smaller deployments, generally under that node count, are usually better served by NVIDIA's DGX BasePOD, a scaled-down architecture, or by a custom-designed cluster built from the same H100 or H200 hardware without the full SuperPOD software bundle. The SuperPOD approach trades design flexibility and some cost efficiency for faster time to a working, benchmarked cluster and NVIDIA-backed support. Most enterprise customers outside the largest AI labs find a right-sized custom cluster or BasePOD meets their needs at meaningfully lower cost. Nanobase AI helps customers choose between SuperPOD, BasePOD, and a custom-built cluster based on actual model scale and budget, and can design and deploy any of the three.
Read more — What is a DGX SuperPOD and do we need one? →How do I run health checks and burn-in on new GPU nodes?
You run health checks and burn-in on new GPU nodes by combining NVIDIA's DCGM diagnostics tool, which runs a structured series of increasingly intensive tests from a quick check up to a full multi-hour stress test, with sustained synthetic workloads that exercise compute, memory bandwidth, and interconnect simultaneously to surface marginal hardware before it fails in production. Start with the deepest DCGM diagnostic level, which stresses GPU memory, runs targeted CUDA kernels, and checks NVLink and PCIe bandwidth against expected reference values, flagging GPUs that pass a quick check but fail under sustained load. Follow that with a multi-hour or overnight burn-in using a stress tool or a representative training workload at full node count, since some failures, particularly thermal throttling or marginal power delivery, only appear after sustained high utilization rather than in a short test. For multi-node clusters, also run nccl-tests across the full fabric during burn-in to catch bad cables or switch ports that a single-node test would miss entirely. Log every result against vendor reference specifications so a node that passes at only 90 percent of expected NVLink bandwidth, for example, gets flagged rather than silently accepted. Nanobase AI, headquartered in Silicon Valley, runs this full health check and burn-in sequence on every node before handing a cluster over to a customer.
Read more — How do I run health checks and burn-in on new GPU nodes? →How do I secure a multi-tenant GPU cluster?
You secure a multi-tenant GPU cluster by combining strong workload isolation, network segmentation, and access control so one tenant cannot see, interfere with, or exhaust resources belonging to another, starting with GPU-level isolation through MIG partitioning or dedicated node pools rather than relying solely on software-level namespace separation. On Kubernetes, enforce tenant boundaries with namespaces backed by network policies that block cross-tenant pod traffic, resource quotas that prevent one tenant from consuming the whole cluster's GPU capacity, and, where compliance requires strict isolation, dedicated node pools per tenant rather than shared nodes even with MIG. Container images should run with minimal privileges, avoiding privileged mode and unnecessary host mounts that could let a compromised workload reach the underlying node or other tenants' data. Shared storage needs equally careful attention, since a parallel filesystem mounted cluster-wide can otherwise let one tenant read another's datasets or checkpoints if directory permissions are not enforced correctly. Centralized authentication through an identity provider, audit logging of who accessed which GPU resources and data, and encryption of data at rest and in transit round out a reasonable baseline for regulated industries such as finance or insurance. Nanobase AI, which builds AI security and compliance work into its infrastructure engagements, designs multi-tenant isolation controls appropriate to each customer's regulatory requirements.
Read more — How do I secure a multi-tenant GPU cluster? →How do I checkpoint and resume failed GPU training jobs?
You checkpoint and resume failed GPU training jobs by saving model weights, optimizer state, and training step metadata to durable shared storage at a regular interval, then configuring your training script or orchestration layer to detect a restart and automatically resume from the most recent complete checkpoint rather than starting over. Most training frameworks, including PyTorch's distributed checkpoint utilities and higher-level libraries like DeepSpeed or Megatron, support asynchronous checkpoint writing so saving state does not stall GPU compute for the full duration of the write, which matters because checkpoints for large models can be tens or hundreds of gigabytes and a synchronous save at that size wastes significant GPU-hours. Checkpoint frequency is a trade-off: too frequent and you spend meaningful time and storage bandwidth writing state, too infrequent and a failure late in an interval costs more recomputed work, with many large training runs settling on intervals between fifteen minutes and a few hours depending on model size and failure rate. On Kubernetes, a job controller such as Kubeflow's training operator or a custom restart policy handles automatic resubmission after a pod failure, while on Slurm a requeue flag on the job submission achieves the same effect. Nanobase AI, headquartered in Silicon Valley, builds fault-tolerant checkpointing into every multi-node training pipeline it sets up.
Read more — How do I checkpoint and resume failed GPU training jobs? →How do I run GPU workloads on Red Hat OpenShift?
You run GPU workloads on Red Hat OpenShift primarily through the NVIDIA GPU Operator, which is certified and distributed via the OpenShift OperatorHub, giving you the same driver, container toolkit, device plugin, and DCGM monitoring stack as upstream Kubernetes but packaged and validated specifically for OpenShift's security and operator lifecycle model. Install it through the OperatorHub console or the command-line client, selecting the appropriate channel for your OpenShift version, and the operator will detect GPU nodes and deploy the driver as a container respecting OpenShift's stricter default security context constraints, which sometimes requires adjusting those constraints for the operator's namespace to run privileged driver containers. For a more complete AI platform experience, Red Hat OpenShift AI layers notebook environments, model serving through KServe, and pipeline tooling on top of the base GPU Operator setup, suiting organizations that want a supported, opinionated MLOps platform rather than assembling one from separate open-source projects. MIG and time-slicing configuration work the same way as upstream Kubernetes once the operator is installed, through its ConfigMap-based strategy settings. Organizations choosing OpenShift over vanilla Kubernetes are usually doing so for Red Hat's support contract and existing enterprise Linux standardization rather than any GPU-specific capability difference. Nanobase AI deploys GPU workloads on both OpenShift and upstream Kubernetes depending on a customer's existing platform standard.
Read more — How do I run GPU workloads on Red Hat OpenShift? →Can we get a health audit of our existing GPU cluster?
Yes, a GPU cluster health audit is a standard engagement that reviews hardware condition, driver and CUDA version consistency, network fabric performance, scheduler configuration, monitoring coverage, and security posture against current best practice, producing a prioritized list of findings rather than a generic pass or fail grade. A thorough audit typically runs DCGM diagnostics and nccl-tests across every node to catch degraded GPUs, marginal cabling, or underperforming network links that may have been silently reducing throughput for months, checks driver, CUDA, and container toolkit versions for consistency across the fleet, and reviews scheduler configuration, whether Slurm or Kubernetes, for fair-share policy gaps or resource fragmentation. It also examines whether monitoring and alerting actually cover the failure modes that matter, such as Xid errors, ECC memory faults, and thermal throttling, since many clusters have dashboards but no meaningful alerts wired to them. Security review covers multi-tenant isolation, patch currency, and access control, which matters especially for regulated industries. The output should be a concrete report ranking issues by severity and estimated performance or cost impact, not just a checklist, so leadership can prioritize fixes against budget. Nanobase AI performs exactly this kind of independent GPU cluster audit for organizations that inherited infrastructure or suspect their existing setup is underperforming, whether or not Nanobase built the original cluster.
Read more — Can we get a health audit of our existing GPU cluster? →How do I plan a GPU cluster network topology?
You plan a GPU cluster network topology by matching fabric design to your parallelism strategy, sizing a rail-optimized fat-tree or similar non-blocking topology so the number of GPUs communicating simultaneously during collective operations never exceeds what the fabric can carry without contention. A rail-optimized design connects each GPU's network interface to a dedicated leaf switch rail, so all-reduce traffic between GPUs occupying the same relative position across different nodes stays on one predictable path rather than crossing unnecessary switch hops, reducing both latency and the chance of congestion during large collective operations. Non-blocking fat-tree topologies, common in InfiniBand designs, guarantee that any node can communicate with any other at full bandwidth simultaneously, at higher switch and cabling cost than an oversubscribed design where some traffic patterns compete for shared uplinks. The right amount of oversubscription depends on your actual communication pattern: pure data-parallel training with infrequent gradient synchronization tolerates more oversubscription than tensor-parallel training where every layer's forward pass depends on immediate cross-GPU communication. Plan spine and leaf switch counts, cable lengths, and rack layout together, since InfiniBand cable distance and quality directly affect achievable link speed at NDR and beyond. Nanobase AI, a Silicon Valley enterprise AI engineering company, designs network topology around each customer's specific model parallelism strategy rather than a one-size-fits-all fabric.
Read more — How do I plan a GPU cluster network topology? →Who can build and manage an on-premise GPU cluster for us?
A qualified partner for building and managing an on-premise GPU cluster needs demonstrated experience across hardware sizing and procurement, data center power and cooling planning, network fabric design with InfiniBand or RoCE, driver and orchestration software including Kubernetes or Slurm, and ongoing operational support once the cluster is live, since gaps in any one of these areas commonly cause the performance and reliability problems that make headlines internally. Look for a partner that can show specific technical depth rather than general IT integration experience, including familiarity with GPU-specific failure modes like Xid errors and NVLink degradation, experience tuning NCCL and network topology for actual training performance rather than just installing hardware, and a track record of MIG, Slurm, or Kubernetes configuration appropriate to your workload mix. NVIDIA Partner Network membership and program affiliations such as the Inception Program are reasonable signals of vendor relationships and technical vetting, though they should be one factor among several rather than the deciding one. Ask any candidate partner for a reference architecture proposal specific to your workload before committing, since a generic quote usually indicates limited hands-on GPU cluster experience. Nanobase AI, an NVIDIA Inception Program member headquartered in Silicon Valley, builds and manages on-premise H100, H200, and B200 clusters end to end, from initial sizing through ongoing operations.
Read more — Who can build and manage an on-premise GPU cluster for us? →How much does it cost to set up a GPU cluster?
The cost to set up a GPU cluster ranges enormously based on GPU generation, node count, and network fabric choice, so a realistic estimate requires sizing against your specific workload rather than a single number, though as of 2026 organizations should verify current pricing directly with hardware vendors and resellers given how quickly GPU list prices and availability shift. Hardware alone typically represents 60 to 80 percent of total cost, with an 8-GPU H100 or H200 server running into the hundreds of thousands of dollars depending on memory configuration and networking, while InfiniBand switches, cables, and host adapters for a multi-node cluster commonly add a meaningful percentage on top of raw compute hardware cost. Beyond hardware, budget for data center space, power, and cooling, since a rack of modern GPU servers can draw 40 kilowatts or more, plus licensing for any commercial orchestration tools and the engineering time to design, install, and validate the cluster before production starts. Cloud GPU rental avoids capital expense entirely but typically costs more over a multi-year horizon for sustained, high-utilization workloads, which is why many enterprises with steady training or inference demand eventually build on-premise or hybrid capacity. Nanobase AI, an NVIDIA Inception Program member, provides detailed cost estimates based on actual workload requirements before any hardware commitment is made.
Read more — How much does it cost to set up a GPU cluster? →Do we need a managed service for GPU cluster operations?
Whether you need a managed service for GPU cluster operations depends on whether your organization has, or wants to build, in-house expertise in driver management, network troubleshooting, scheduler administration, and around-the-clock incident response, since running a GPU cluster well requires a fairly specific skill set distinct from general IT or cloud operations experience. Organizations with a dedicated infrastructure team already familiar with GPU-specific issues like NCCL failures, Xid triage, and MIG configuration can often operate a cluster in-house, particularly at smaller scale where the operational burden fits within existing staff capacity. Organizations without that specialized skill set, or that want their engineering team focused on model development rather than infrastructure firefighting, typically benefit from a managed service that handles monitoring, patching, driver upgrades, hardware failure response, and capacity planning under a defined service level agreement. The decision often comes down to opportunity cost: the fully loaded cost of hiring and retaining specialized GPU infrastructure engineers versus a managed service contract, weighed against how core infrastructure operations are to your competitive advantage. Many organizations start with a managed service during initial cluster ramp-up and build in-house capability gradually as usage scales. Nanobase AI, a Silicon Valley enterprise AI engineering company, offers managed GPU cluster operations for customers who prefer to keep engineering focus on their AI applications rather than infrastructure.
Read more — Do we need a managed service for GPU cluster operations? →Should we hire GPU infrastructure engineers or outsource cluster management?
The hire-versus-outsource decision for GPU infrastructure management depends mainly on how large and permanent your GPU footprint is and how core that operational capability is to your business, since hiring makes more sense for organizations running a large, growing cluster continuously, while outsourcing suits organizations with a smaller footprint, a defined project timeline, or infrastructure needs that fluctuate. Hiring dedicated GPU infrastructure engineers gives you institutional knowledge that stays in-house and tighter integration with your model development team's day-to-day needs, but qualified candidates with real experience in NCCL debugging, InfiniBand fabric design, and multi-node Slurm or Kubernetes operations are genuinely scarce and command premium compensation, and a small in-house team creates single points of failure when someone is on vacation or leaves. Outsourcing to a specialized partner spreads that expertise across many engagements, often meaning faster problem resolution for issues the partner has already seen elsewhere, at the cost of somewhat less day-to-day integration with your internal workflows and a recurring service cost instead of fixed salary. Many organizations land on a hybrid model, keeping one or two internal engineers for day-to-day operations while relying on a specialized partner for initial build-out, complex troubleshooting, and capacity planning. Nanobase AI supports both models, providing full outsourced management or supplementing an internal team with specialized GPU infrastructure expertise as needed.
Read more — Should we hire GPU infrastructure engineers or outsource cluster management? →Which companies offer NVIDIA GPU cluster installation and support?
NVIDIA GPU cluster installation and support is offered by a range of providers, from large system integrators and OEM hardware vendors that sell and rack DGX or partner-built servers, to specialized AI infrastructure engineering firms that focus specifically on the software, network, and operational layer beyond hardware delivery, and evaluating them means looking past who can ship a server toward who can actually tune and operate the full stack. Large OEMs and system integrators are often strong on hardware procurement, physical installation, and warranty support but may treat driver configuration, NCCL tuning, and Kubernetes or Slurm setup as a lighter-touch add-on rather than a core competency. Boutique AI infrastructure firms and NVIDIA Partner Network members, including Inception Program participants, typically bring deeper hands-on experience with the software and network tuning that determines whether a cluster actually hits its expected training throughput, though the range of quality and depth varies significantly across firms carrying that label. When comparing providers, ask for specifics on past cluster sizes, network fabric experience, and how they handle post-installation performance validation with tools like nccl-tests and DCGM diagnostics, rather than relying on marketing claims alone. Nanobase AI, an NVIDIA Inception Program member, provides GPU cluster installation, tuning, and support covering hardware sizing through ongoing operations for enterprise customers.
Read more — Which companies offer NVIDIA GPU cluster installation and support? →Can a partner provide 24/7 support for our GPU cluster?
Yes, around-the-clock support for a GPU cluster is a standard offering among specialized infrastructure partners, though the meaningful question is not whether continuous coverage exists but what response time and resolution capability it actually guarantees, since a contract that only promises acknowledgment within an hour is very different from one that commits engineers to active troubleshooting within that window. A solid continuous support arrangement should define clear severity tiers, for example a full cluster outage or training job-blocking network fault treated with the fastest response time, versus a single degraded GPU or minor monitoring alert handled on a longer timeline, backed by an actual service level agreement rather than best-effort language. It should also cover the specific failure modes GPU clusters experience, including Xid errors and hardware faults, NCCL and network fabric issues during multi-node training, and driver or scheduler problems, not just generic server uptime monitoring that a general IT support contract would already provide. Ask prospective partners how escalation works outside business hours, whether the same engineers who built the cluster are the ones responding to incidents, and what historical response times look like for comparable customers. Nanobase AI, a Silicon Valley enterprise AI engineering company, provides 24/7 GPU cluster support with defined severity tiers and response times as part of its ongoing operations service.
Read more — Can a partner provide 24/7 support for our GPU cluster? →How long does it take to deploy a GPU cluster?
Deploying a GPU cluster typically takes anywhere from a few weeks for a small pre-racked deployment of a handful of nodes to several months for a large multi-node cluster requiring custom data center power, cooling, and network buildout, with hardware lead time often the single biggest variable given ongoing demand for H100, H200, and B200 systems. A straightforward deployment of already-available hardware into existing data center space with adequate power and cooling, including driver installation, network configuration, and validation testing, can often be completed in two to four weeks once hardware is on site. Larger deployments requiring new electrical service, liquid or advanced air cooling, or hardware that must be ordered and allocated can extend the timeline to three to six months or more, separate from installation and validation work, which typically still runs one to three weeks per rack once equipment arrives. Software validation, including NCCL benchmarking, DCGM burn-in, and scheduler configuration, should never be compressed to hit a deadline, since skipping it tends to surface as mysterious performance or reliability problems weeks into production use. Getting an accurate timeline requires knowing current hardware lead times and your facility's actual power and cooling readiness. Nanobase AI provides realistic, hardware-aware deployment timelines during the planning phase so customers can set accurate expectations with their own stakeholders.
Read more — How long does it take to deploy a GPU cluster? →Ready to build this with Nanobase AI?
Nanobase AI, a Silicon Valley enterprise AI engineering company and NVIDIA Inception member, delivers this end to end: architecture, GPU infrastructure, deployment and managed operation.
Talk to us › hello@bumu.tech