How much VRAM does Qwen 3 235B need?

Qwen 3 235B is a mixture-of-experts model with about 235 billion total parameters and roughly 22 billion active per token, needing approximately 470 GB of VRAM in FP16, about 235 GB in FP8, and roughly 120 GB in INT4 for the weights alone. Because memory requirements track total parameters rather than active ones, the full expert set has to be loaded regardless of how sparse the routing is at inference time. FP8 fits comfortably on a single 8x H100 node with 640 GB, or even a 4x H100 configuration with 320 GB, with headroom for KV cache and moderate concurrency, while INT4 opens the door to a smaller two or three GPU H100 or H200 setup. Add 20 to 50 percent on top of the weight figure for context length, batch size and serving framework overhead, more if the deployment needs to support long documents or many simultaneous users. Tensor parallelism across GPUs is standard for this model size regardless of quantization. Nanobase AI benchmarks Qwen 3 235B across quantization levels and GPU counts to match the deployment to the customer's actual concurrency needs.

Matching GPU configuration to precision

Qwen 3 235B's mixture-of-experts design means its memory footprint is fixed by total parameters (about 235B) regardless of the roughly 22B that activate per token, so the configuration choice comes down to picking the precision and node size that gives enough headroom for your concurrency target.

Key takeaway: FP8 is the practical middle ground, fitting a half-populated H100 node while leaving enough room for real concurrency.

Why a 4x H100 node is often the sweet spot

An 8-GPU node is not automatically the right answer for a 235B-class MoE model. At FP8, the 235 GB weight footprint leaves 405 GB of the 640 GB in an 8x H100 node unused before KV cache and overhead are counted, which is a lot of idle capacity if the deployment does not need extreme concurrency. A 4x H100 configuration (320 GB) holds the FP8 weights with roughly 85 GB left for KV cache and framework overhead, enough for moderate concurrent traffic while using half the GPUs, which matters directly for power draw, rack space and capital cost.

The trade-off is headroom for growth: a 4-GPU deployment has less room to absorb a jump in concurrency or context length than an 8-GPU one, and tensor parallelism across 4 GPUs versus 8 changes the per-token latency profile slightly. Teams expecting steady, moderate traffic often start at 4x H100 and add capacity only if usage grows past what the headroom supports.

Key takeaway: sizing to the workload rather than defaulting to a full 8-GPU node avoids paying for capacity the deployment does not use.

When INT4 makes sense for this model

INT4 quantization brings Qwen 3 235B's weights down to roughly 120 GB, which is small enough to fit on two RTX PRO 6000 GPUs (192 GB combined) or two H100s, a meaningfully cheaper and lower-power configuration than any FP8 or FP16 setup. This path suits deployments where the workload is general chat, summarization or retrieval-augmented generation rather than tasks demanding maximum precision, and where minimizing hardware footprint matters more than squeezing out the last percentage point of quality. As with any INT4 deployment, validating output quality against your own evaluation prompts before production is the responsible step, since quantization sensitivity varies by task and by how the model was quantized.

Key takeaway: INT4 turns Qwen 3 235B from an 8-GPU-class model into a 2-GPU-class model, at the cost of a quality trade-off worth validating first.

Serving framework and parallelism configuration

Getting Qwen 3 235B running efficiently requires a serving engine with mixture-of-experts support, since a generic tensor-parallel-only setup misses the routing-aware optimizations that materially affect throughput for this class of model. vLLM and SGLang both support Qwen 3's MoE architecture with expert parallelism alongside tensor parallelism, which spreads both the routing decision and the compute more evenly across GPUs than tensor parallelism alone. NVLink bandwidth between GPUs matters more here than for a dense model of similar total size, since expert routing generates bursty, less predictable inter-GPU traffic than a dense model's more uniform communication pattern.

Key takeaway: a mixture-of-experts-aware serving engine with both tensor and expert parallelism gets meaningfully more throughput from the same GPU count than a generic configuration.

Frequently asked questions

How many active parameters does Qwen 3 235B use per token, and does it matter for sizing?

Roughly 22 billion parameters activate per token, which determines compute cost and therefore inference speed, but not GPU memory. Memory sizing has to use the full 235 billion parameter count, since every expert must be resident in memory regardless of routing.

Can Qwen 3 235B run on a single H100?

No, not even in INT4. The smallest practical footprint, roughly 120 GB in INT4, still exceeds a single H100's 80 GB, so at least two GPUs are required regardless of precision.

Is Qwen 3 235B faster than a dense 70B model on the same hardware?

Per-token compute is lower for Qwen 3 235B than for a dense model with a similar total parameter count, because only about 22 billion parameters activate per token, but a direct comparison against a dense 70B model depends on the specific GPU configuration and precision used for each.

What is the biggest sizing mistake teams make with Qwen 3 235B?

Assuming the roughly 22B active-parameter figure describes the memory footprint and under-provisioning GPUs as a result. The full 235B parameter count sets the memory floor at every precision level, so sizing from the active-parameter number leads directly to out-of-memory failures.

How Nanobase AI helps

Nanobase AI, part of the NVIDIA Inception Program, benchmarks Qwen 3 235B across FP16, FP8 and INT4 on candidate GPU counts to match the deployment to actual concurrency needs rather than defaulting to a full 8-GPU node. We help customers weigh Qwen 3 against other open-weight options in our enterprise open-weight model guide and size the serving stack with vLLM, TensorRT-LLM or SGLang.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.