A mixture-of-experts model splits its parameters into many specialized sub-networks, called experts, and a router network selects only a small subset of them to process each token, rather than running every parameter for every input the way a dense model does. This lets DeepSeek V3 hold 671 billion total parameters while activating only about 37 billion per token, or Llama 4 Maverick hold 400 billion total parameters while activating around 17 billion, giving the model much greater overall capacity without a proportional increase in the compute cost of each forward pass. Meta and DeepSeek both adopted this architecture because it lets them scale total model knowledge and specialization further within a fixed inference budget, which is increasingly the binding constraint at frontier scale. The trade-off is memory: even though fewer parameters compute per token, all experts must still be loaded into GPU memory, so a mixture-of-experts model needs more total VRAM than its active-parameter count alone would suggest. This is why serving DeepSeek V3 or Llama 4 Maverick well typically requires multiple high-memory GPUs even though inference is comparatively fast. As an NVIDIA Inception Program member, Nanobase AI sizes GPU clusters specifically around this gap between total and active parameters.
The concept is simple; serving it well is not
The idea behind a mixture-of-experts model, a router sending each token to a small subset of specialized sub-networks, is easy to state, but the infrastructure implications are where teams new to MoE models get surprised. Because a router can theoretically send any token to any expert, every expert must sit in GPU memory ready to be used, and in practice, load across experts is rarely perfectly even, which affects both memory planning and throughput in ways a dense model never requires.
Understanding the concept explains why MoE models are efficient; understanding the serving mechanics explains why they still need substantial GPU infrastructure.
Dense vs MoE: what actually changes at serving time
| Aspect | Dense model | Mixture-of-experts model |
|---|---|---|
| Memory needed | Scales with total parameters only | Scales with total parameters, same as dense at equal total size |
| Compute per token | Scales with total parameters | Scales with active parameters only, much lower |
| GPU topology | Simpler tensor parallelism | Often needs expert parallelism across GPUs or nodes |
| Load balancing | Not applicable | Router load imbalance can bottleneck specific GPUs |
| Communication overhead | Lower | All-to-all communication between experts adds network load |
A mixture-of-experts model trades higher communication and topology complexity for a large compute-per-token saving, which is worth it at scale but is not a free efficiency gain.
Expert parallelism changes your cluster design
Serving a large MoE model like DeepSeek V3 or Llama 4 Maverick efficiently typically means spreading experts across multiple GPUs, a technique called expert parallelism, which requires high-bandwidth interconnects between GPUs, ideally NVLink within a node and InfiniBand across nodes, since tokens must be routed to whichever GPU holds the relevant expert for every forward pass. This is a materially different cluster design requirement than serving a same-size dense model, where straightforward tensor parallelism across GPUs is usually sufficient.
Plan interconnect bandwidth, not just total GPU memory, when sizing infrastructure for a large mixture-of-experts model; the router's communication pattern is a real bottleneck if under-provisioned.
Router imbalance is a real operational concern
In production, certain experts can receive disproportionately more tokens than others depending on the traffic mix, creating a bottleneck on the GPUs hosting those popular experts even while others sit underutilized. Serving frameworks that support MoE models, including vLLM and TensorRT-LLM, include load-balancing strategies to mitigate this, but monitoring actual per-expert utilization in production remains worthwhile, since imbalance that looks fine in testing can shift once real traffic patterns emerge.
Monitor per-expert GPU utilization in production, not just aggregate throughput, since router imbalance can silently degrade performance even when overall metrics look healthy.
Frequently asked questions
Does a mixture-of-experts model need fewer GPUs than a dense model of the same total size?
No, for memory purposes it needs the same amount, since all experts must be resident in GPU memory regardless of activation pattern. The savings from MoE architecture show up in compute cost and inference speed per token, not in total memory footprint.
Can a single GPU serve a large MoE model like DeepSeek V3?
Not the full model; DeepSeek V3's roughly 671 billion total parameters require far more memory than any single current GPU provides, so multi-GPU deployment with expert parallelism is required regardless of how few parameters activate per token.
Is expert parallelism harder to set up than standard tensor parallelism?
Generally yes, since it requires the serving framework to route tokens dynamically between GPUs based on the router's decisions, rather than a fixed data split. Frameworks like vLLM and TensorRT-LLM handle much of this automatically, but interconnect bandwidth still needs to be provisioned correctly.
How Nanobase AI helps
As an NVIDIA Inception Program member, Nanobase AI sizes GPU clusters specifically around the gap between total and active parameters in mixture-of-experts models, including interconnect topology and expert-parallel serving configuration. See our H100 vs H200 vs B200 inference comparison or the related question on Llama 4 versus Qwen 3 infrastructure fit. Explore our solutions.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.