How do I enable MIG on an H100 in Kubernetes?

You enable MIG on an H100 in Kubernetes by first switching the physical GPU into MIG mode with an nvidia-smi command on the node, which requires a GPU reset or reboot, and then configuring the NVIDIA GPU Operator to apply a partitioning profile through its ConfigMap. The operator supports two strategies, single, where every GPU on a node uses the same partition layout, and mixed, where different GPUs on the same node can run different profiles, useful when a node has multiple H100s serving different workload sizes. You select a profile such as all-1g.10gb for seven small inference instances or all-3g.40gb for three medium instances, apply the corresponding MIG configuration label to the node, and the operator's MIG manager reconfigures the GPU without manual intervention. Kubernetes then exposes each instance as a separate schedulable resource, for example nvidia.com/mig-1g.10gb, so pods request a slice rather than a whole GPU. It is worth validating with an nvidia-smi listing afterward to confirm the expected instance count and memory sizes appeared. Nanobase AI configures MIG profiles as part of its Kubernetes GPU Operator deployments so customers get the right slice size for each workload from day one.

Step-by-step: enabling MIG through the GPU Operator

1. Confirm the target node's GPUs are Ampere-generation or newer and currently unpartitioned by checking nvidia-smi -L output.
2. Label the node with the desired MIG configuration, for example nvidia.com/mig.config=all-3g.40gb, which the operator's MIG Manager watches for.
3. Cordon the node before applying the label if workloads are currently running, since the reconfiguration requires a GPU reset.
4. Let the MIG Manager apply the profile; it resets the GPU into MIG mode and partitions it according to the label, which can take a couple of minutes per GPU.
5. Confirm the new MIG instances appear as separate schedulable resources, for example nvidia.com/mig-3g.40gb, by describing the node.
6. Uncordon the node and submit a test pod requesting one MIG slice to confirm scheduling works end to end.

kubectl label node <node-name> nvidia.com/mig.config=all-3g.40gb --overwrite
kubectl describe node <node-name> | grep mig

Skipping the cordon step before a MIG reconfiguration is the most common way to disrupt a running job unintentionally, since the GPU reset happens regardless of whether something is using it.

Single vs mixed strategy

The single strategy is simpler to reason about and is the default most clusters start with. mixed earns its added complexity once a node genuinely hosts, for example, one H100 partitioned into seven 1g.10gb instances for small models next to another H100 left as 7g.80gb for a larger one.

Verifying MIG instances are actually schedulable

After the MIG Manager reports success, kubectl describe node should list the MIG profile resources, such as nvidia.com/mig-1g.10gb: 7, under Allocatable rather than the generic nvidia.com/gpu resource. A pod spec then requests nvidia.com/mig-1g.10gb: 1 instead of a whole GPU. Running nvidia-smi -L from inside a scheduled pod should show only the assigned instance's memory and compute slice, not the full physical GPU, confirming isolation is actually in effect rather than just labeled.

Common issues and fixes

A MIG configuration label that never takes effect usually means the node needs a reboot the operator could not perform automatically, often because a workload was still pinned to the GPU during the reconfiguration attempt. A MIG Manager pod stuck in a pending state for an extended period almost always means the GPU reset failed silently and the node needs manual intervention, typically a full reboot rather than repeated label reapplication. If MIG resources never appear as allocatable even after a successful reset, check that the device plugin's ConfigMap has MIG_STRATEGY set consistently with the label applied, since a mismatch between the two leaves instances invisible to the scheduler.

Frequently asked questions

Does enabling MIG on an H100 require a full node reboot?

Often yes, or at minimum a GPU-level reset, since the driver must reinitialize the GPU's internal partitioning. The exact requirement depends on driver version, so test the reconfiguration on one canary node before scripting it across a fleet. Budget for a brief scheduling gap on that node either way, since workloads must be drained before the reset regardless of whether it escalates to a full reboot.

Can I resize a MIG profile without deleting existing workloads first?

No. Any workload still using GPU or MIG resources on that node must be evicted before the GPU can be reset into a new profile, which is why cordoning the node ahead of a label change is a required step, not an optional precaution.

How do I roll back to an unpartitioned GPU?

Set the node's MIG configuration label to a value indicating MIG should be disabled, which the operator interprets the same way as any other profile change: a GPU reset that reverts the card to full, unpartitioned mode. As with any MIG reconfiguration, drain and cordon the node first, since any workload still holding a MIG resource will block the reset from completing cleanly.

Does the MIG Manager work the same way on A100 as on H100?

Yes, the mechanism is identical since both are MIG-capable Ampere and Hopper generation GPUs; only the specific profile names and instance counts differ because A100 and H100 have different total memory and compute unit counts. The GPU Operator's configuration labels and reconciliation logic apply uniformly across both, so a fleet mixing A100 and H100 nodes can use the same MIG Manager deployment without separate handling.

How Nanobase AI helps

Nanobase AI configures MIG profiles as part of its Kubernetes GPU Operator deployments, choosing single or mixed strategy based on actual workload mix and validating that partitioned instances are correctly schedulable before handing a cluster over. For the underlying sizing decision, see NVIDIA MIG profiles and sizing or explore our platform.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.