How much memory does a MoE model need compared to its active parameters?

A mixture-of-experts model needs GPU memory sized to its total parameter count, not its active parameter count, because every expert has to be loaded into memory even though the routing mechanism only sends each token through a small subset of experts during computation. This is why a model like DeepSeek with 671 billion total parameters but only about 37 billion active per token still needs roughly the same memory footprint as a dense model with 671 billion parameters, even though its inference compute cost per token looks much closer to a 37 billion parameter dense model. Active parameters primarily determine inference speed and FLOPs per token, which is why mixture-of-experts models can generate tokens faster than a dense model of equivalent total size, but they do not reduce the hardware needed to hold the model in memory in the first place. This distinction is the single most common source of confusion when sizing hardware for mixture-of-experts models like DeepSeek, Qwen 3's larger variants, or Llama 4 Maverick, since teams sometimes assume active parameter count alone determines the GPU requirement. Nanobase AI sizes mixture-of-experts deployments against total parameter count first, then optimizes throughput around the active parameter compute profile.

The distinction that trips up most first-time MoE sizing

A mixture-of-experts (MoE) architecture routes each token through only a small subset of its expert layers, which is why active parameter count is often the headline number quoted for these models' inference speed. But routing decisions are made per token and can select different experts for different tokens in the same request, which means every expert has to be resident in GPU memory ready to be selected at any moment. There is no way to predict in advance which experts a given request will need, so none of them can be safely left out of memory, and that single fact is what ties GPU memory sizing to total parameters rather than active ones.

Comparing total and active parameters across common MoE models

Reading this table by the wrong column is the most common MoE sizing mistake: assuming a model needs GPU memory proportional to its active parameter figure would suggest DeepSeek R1 needs roughly the same hardware as a 37B dense model, when the real memory requirement is closer to a 671B dense model, a difference of well over 15 times.

Why MoE models are still worth the memory cost

The payoff for holding all those experts in memory is inference speed and, often, quality per unit of compute. A mixture-of-experts model can generate tokens meaningfully faster than a dense model of equivalent total size, because the actual FLOPs per token track the smaller active parameter count, not the full model. This is why organizations choose large MoE models like DeepSeek R1 or Llama 4 Maverick over similarly-sized dense alternatives when they have the memory to spare: the memory cost is fixed and upfront, but the throughput benefit is realized on every single request served afterward.

Practical implications for cluster sizing

1. Size the GPU cluster to total parameters first, using the same precision-based weight formula (FP16, FP8, INT4) applied to a dense model, then treat active parameters as a separate throughput-planning input.
2. Expect better tokens-per-second than a dense model of the same total size once memory is provisioned, which can translate into serving more concurrent users per GPU-hour than the raw model size would suggest.
3. Consider expert parallelism for very large MoE models, distributing different experts across different GPUs rather than replicating the full model on each, which is a distinct technique from the tensor parallelism used for dense models and requires serving engine support (available in TensorRT-LLM and vLLM for major MoE architectures).
4. Do not assume a smaller active-parameter figure means a smaller GPU bill; procurement conversations that start from the active parameter number alone will undersize the cluster.

Frequently asked questions

Does quantization work the same way on MoE models as dense models?

Yes, the same FP16, FP8 and INT4 precision reductions apply to MoE expert weights, and the total-parameter-based memory formula still governs sizing at the reduced precision, so an INT4 DeepSeek R1 still needs roughly a quarter of its FP16 total-parameter footprint, not its active-parameter footprint.

Why do MoE models exist if they need so much memory?

Because the memory cost is a one-time infrastructure investment while the compute savings apply to every token generated afterward, MoE architectures let model builders scale total capacity and quality further than a dense model would allow for the same per-token inference cost.

Is DeepSeek R1's hardware requirement really the same as a 671B dense model?

Essentially yes for memory purposes, as detailed in GPUs needed for DeepSeek R1 671B on-premise; the practical difference from a genuinely dense 671B model shows up in inference speed and cluster utilization, not in the GPU memory bill.

Can experts be selectively loaded to save memory?

Some inference frameworks support offloading rarely-used experts to CPU RAM, but this reintroduces the latency penalty of CPU memory bandwidth on any request that happens to route to an offloaded expert, making it a niche technique for memory-constrained setups rather than a standard production pattern.

How Nanobase AI helps

Nanobase AI sizes mixture-of-experts deployments against total parameter count first, then optimizes throughput around the active parameter compute profile, avoiding the common mistake of under-provisioning memory based on a model's advertised active-parameter figure alone.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.