How many GPUs do I need to run Llama 4 Maverick?

Llama 4 Maverick is a mixture-of-experts model with about 400 billion total parameters and roughly 17 billion active per token, and because every expert has to be resident in memory even though only a subset activates per token, the full weight set needs around 400 GB in the FP8 format it ships in. That rules out a single GPU and typically points to one 8x H100 node with 640 GB or an 8x H200 node with 1,128 GB as the practical minimum, with the H200 configuration leaving substantially more headroom for KV cache, batching and multiple concurrent users. Tensor parallelism across the node is required regardless of GPU choice, and NVLink bandwidth between GPUs matters more here than with dense models because of the routing overhead in mixture-of-experts inference. Quantizing further to INT4 can reduce the footprint enough to consider a smaller cluster, but with some accuracy trade-off on a model already using a sparse architecture. Serving frameworks like vLLM and TensorRT-LLM both support Llama 4's architecture with continuous batching for production throughput. Nanobase AI, an NVIDIA Inception Program member, designs and installs the multi-GPU clusters this class of mixture-of-experts model requires, from GPU selection through InfiniBand networking.

Building the exact node memory budget

Knowing that Maverick needs about 400 GB of FP8 weights is only the starting point; the number that determines whether a deployment actually works in production is what remains after weights are loaded. The table below breaks down a realistic budget for both common node choices at moderate concurrency.

Key takeaway: both nodes hold the weights, but the H200 node leaves roughly three times more room for KV cache and concurrent users.

Why total parameters, not active parameters, set the floor

Maverick's headline efficiency, about 17 billion active parameters per token out of roughly 400 billion total, describes compute cost per token, not memory. Every expert in a mixture-of-experts model has to sit resident in GPU memory because the router can select any of them for any token, so the full 400 billion parameters have to be loaded regardless of how sparse the routing pattern is at inference time. This is the single most common misunderstanding in sizing MoE models: teams see the small active-parameter count and assume the memory footprint is closer to a 17B dense model, when it behaves like a 400B model for memory and like a much smaller model for compute.

The practical consequence is that Maverick needs the same GPU memory tier as a dense model of similar total size, while actually generating tokens faster than a dense 400B model would, because less compute runs per token. Sizing has to account for both facts at once.

Key takeaway: budget GPU memory for the full expert set, not the active-parameter count, but expect compute cost closer to a much smaller dense model.

Deployment steps for a Maverick node

1. Confirm the checkpoint precision. Maverick ships natively in FP8, so plan around the ~400 GB figure rather than converting to FP16 first.
2. Choose the node: 8x H100 works for FP8 with tight KV cache headroom; 8x H200 is the safer choice for any meaningful concurrency or long context.
3. Configure tensor parallelism, and expert parallelism if the serving engine supports it, across all 8 GPUs; a single node's NVLink domain is what makes the expert-routing communication pattern practical.
4. Verify InfiniBand or NVLink bandwidth between GPUs before going live, since MoE routing generates more inter-GPU traffic than a dense model of similar size.
5. Load-test with realistic concurrent sessions before committing to a fixed context length limit in production.

Key takeaway: an 8-GPU node with fast intra-node interconnect is the deployment unit for Maverick, not a smaller server scaled up later.

Rack power and the INT4 alternative

An 8x H100 or H200 server drawing 10 to 11 kW at full load exceeds what many enterprise racks were built for, so confirming power and cooling capacity before ordering a Maverick-class node is as important as the memory arithmetic itself. Teams constrained on rack power, or wanting to avoid a full 8-GPU node altogether, can quantize Maverick to INT4, which shrinks the roughly 400 GB FP8 footprint enough to fit a smaller GPU count, at the cost of an accuracy trade-off on a model that is already sparse by design and worth validating carefully on your own reasoning and instruction-following tasks before committing.

Key takeaway: rack power is a real constraint for a Maverick-class node, and INT4 is the lever for teams that cannot provision a full 8-GPU server.

Frequently asked questions

Can Llama 4 Maverick run on fewer than 8 GPUs?

Not practically at FP8. The roughly 400 GB weight footprint already exceeds four H100s (320 GB) and even a 4x H200 configuration (roughly 564 GB) leaves very little for KV cache. INT4 quantization can shrink the footprint enough to consider smaller configurations, but with an accuracy trade-off that needs its own validation.

Is Llama 4 Scout a lighter alternative to Maverick?

Yes. Scout is a smaller Llama 4 variant with far fewer total parameters, aimed at deployments that do not need Maverick's full capacity. It fits meaningfully smaller GPU configurations, making it worth evaluating first if a single H100 node with headroom is the target.

Does Maverick need InfiniBand if it fits on one node?

Within a single 8-GPU node, NVLink handles GPU-to-GPU communication, not InfiniBand. InfiniBand only becomes necessary if the deployment spans multiple nodes, for example to add capacity beyond what one node's KV cache headroom allows.

How does Maverick's throughput compare to a dense 400B model on the same hardware?

Because only about 17 billion parameters activate per token, Maverick generates tokens with meaningfully less compute per step than a dense model of similar total size, which typically translates into higher tokens-per-second throughput on identical hardware, even though the memory requirement is comparable.

How Nanobase AI helps

Nanobase AI, an accepted member of the NVIDIA Inception Program, designs and installs the 8-GPU clusters that mixture-of-experts models like Llama 4 Maverick require, from H100 or H200 selection through NVLink and InfiniBand topology and tensor and expert parallel configuration in vLLM or TensorRT-LLM. We benchmark the actual model on candidate hardware before a purchase is made, and compare it against alternatives in our open-weight model guide. See our solutions for the full deployment stack.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.