What is NVIDIA MIG and when should we use it?

NVIDIA Multi-Instance GPU, or MIG, is a hardware feature on A100, H100, and H200 GPUs that partitions a single physical GPU into up to seven fully isolated instances, each with its own dedicated memory, cache, and compute cores. Use MIG when workloads are individually small relative to the GPU, such as inference for models under roughly 10 to 20 billion parameters, batch data preprocessing, or notebook development, where giving one user an entire 80 GB or 141 GB GPU wastes capacity. Each MIG instance gets guaranteed quality of service with no interference from other instances, unlike time-slicing, which matters for latency-sensitive inference SLAs. MIG is not suitable for large distributed training jobs that need the full memory and bandwidth of a GPU, or for workloads that need NVLink between instances on the same card. Typical MIG profiles split an H100 into instances ranging from 1g.10gb up to 7g.80gb, and the GPU Operator can apply a chosen profile automatically across a node pool. Nanobase AI sizes GPU clusters with MIG partitioning built in where it improves utilization, and without it where raw training performance is the priority.

The H100 MIG profile reference

MIG partitions a physical GPU along both compute and memory dimensions, expressed as a profile name like 3g.40gb, meaning three compute slices and 40 GB of memory. An H100 80 GB supports these standard profiles.

A GPU running seven 1g.10gb instances gives seven users guaranteed isolated capacity, but each instance has only a fraction of the full GPU's memory bandwidth and compute, so profile choice is a direct trade-off between instance count and per-instance performance. A100 80 GB supports an equivalent profile family scaled to its own compute and memory totals.

Compute instances vs GPU instances

MIG partitioning happens in two layers that are easy to conflate. A GPU Instance (GI) reserves a slice of memory and cache along with its own memory controllers, giving hardware-level isolation from other GIs on the same card. Within a GI, one or more Compute Instances (CIs) divide the streaming multiprocessors, letting a single GI optionally be shared by multiple compute-only workloads that still see the same reserved memory. Most inference deployments use one CI per GI for simplicity, but the two-layer model exists for cases where several lightweight processes need to share compute within a memory-isolated boundary, such as a multi-process serving setup for a small model.

Sizing a profile to the actual workload

Pick a profile by starting from the memory footprint of the model plus its KV cache headroom, not from a desire to maximize instance count. A model needing roughly 15 GB of memory including cache fits comfortably in a 2g.20gb instance, while a model needing closer to 35 GB needs 3g.40gb and only two such instances fit per H100. Oversubscribing memory forces smaller batch sizes or shorter context windows than the same model would get on an ungated GPU, so the profile decision is really a latency-and-throughput decision wearing a memory-sizing disguise.

1. Estimate model memory plus KV cache headroom at your expected concurrency and context length.
2. Match that figure to the smallest MIG profile that comfortably fits it, leaving margin for activation memory.
3. Multiply by the number of instances the profile allows per GPU to get total serving capacity per card.
4. Validate with a real load test before committing the profile fleet-wide, since KV cache growth under concurrent load is easy to underestimate.

Where MIG is the wrong tool

MIG is not suitable for distributed training or any workload that needs the full memory and bandwidth of a GPU, since a partitioned instance cannot access memory outside its own slice and there is no NVLink connectivity between MIG instances on the same card. Large fine-tuning jobs, any tensor-parallel workload, and anything sensitive to full-GPU memory bandwidth should run on unpartitioned GPUs instead. Reaching for MIG on a training cluster because "it improves utilization" almost always backfires, since training throughput drops far more than the utilization gain is worth.

Frequently asked questions

Can I change a GPU's MIG profile without rebooting the node?

Reconfiguring MIG mode typically requires a GPU reset, and on some driver versions a full node reboot, since the GPU must reinitialize its instance layout. The NVIDIA GPU Operator's MIG Manager can automate this but the reset step itself is unavoidable.

Does every NVIDIA GPU support MIG?

No. MIG requires Ampere-generation or newer data center GPUs such as A100, H100, and H200. Consumer and lower-tier data center cards like L40S, L4, and A10 do not support MIG and rely on time-slicing instead for GPU sharing. Check the specific SKU against NVIDIA's MIG support matrix before planning a deployment around it, since generation alone does not guarantee every card in a family carries the feature.

Can different GPUs on the same node run different MIG profiles?

Yes, using the GPU Operator's "mixed" MIG strategy, which lets each GPU on a node carry its own profile rather than forcing every GPU on that node to match. This suits nodes serving a mix of small and medium-sized models.

Does MIG add measurable latency overhead compared to a full GPU?

Each MIG instance runs with hardware-isolated resources and no software-level virtualization overhead beyond the fixed partition, so latency for a workload that fits comfortably in its slice is close to what the same fraction of an unpartitioned GPU would deliver.

How Nanobase AI helps

Nanobase AI sizes MIG profiles against actual model memory footprints and traffic patterns rather than defaulting to maximum instance count, validating each profile choice with a real load test before it goes into production. For a full comparison against the alternative GPU-sharing approach, see MIG vs time-slicing for GPU sharing, and for hardware specifics see H100 vs H200 vs B200 for LLM inference.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.