The right GPU configuration for Qwen 3 serving 200 users depends heavily on which Qwen 3 variant is chosen: for the dense Qwen 3 32B model in FP8, needing about 32 GB of weights, two H100 GPUs with tensor parallelism comfortably cover the realistic peak concurrency for 200 users, typically a smaller fraction of that total active at once, while also providing redundancy against a single GPU failure. For the larger Qwen 3 235B mixture-of-experts model, needing roughly 235 GB in FP8, a single 8x H100 node or a 4x H100 configuration with careful KV cache budgeting is the more realistic starting point, trading higher hardware cost for meaningfully better model quality on complex tasks. Two hundred total users rarely means two hundred simultaneous requests, so basing the configuration on measured or estimated peak concurrency rather than headcount avoids over-provisioning for a load that never actually occurs. Average conversation length and whether the deployment includes document retrieval both shift the KV cache budget and should be factored into the final GPU count. Nanobase AI configures and benchmarks the specific Qwen 3 variant against a customer's expected concurrency before finalizing a server specification.
The Qwen 3 family spans two very different sizing conversations
Qwen 3 is not one sizing decision but several, since the family includes dense models from single-digit billions of parameters up through 32B, alongside a large mixture-of-experts variant with 235B total parameters. Two hundred users tells you almost nothing about GPU count until the model variant is chosen, because the dense and MoE variants land in completely different hardware tiers. The variant decision, driven by required output quality for the target tasks, should come before the GPU count conversation, not after.
Server configuration by variant
| Qwen 3 variant | Approx. size | Precision | Configuration for 200-user peak concurrency |
|---|---|---|---|
| Smaller dense (up to ~14B) | ~14B or less | FP8 | 1 GPU (H100, H200, or RTX PRO 6000), substantial headroom |
| Dense 32B | ~32B | FP8 (~32 GB) | 2x H100 with tensor parallelism, comfortable headroom plus redundancy |
| MoE 235B (total params) | ~235B total, ~22B active | FP8 (~235 GB) | 4x H100 with careful KV cache budgeting, or a full 8x H100 node for comfortable headroom |
The 32B dense row is the more common production choice for a 200-user internal deployment prioritizing cost efficiency, while the 235B MoE row trades meaningfully higher hardware cost for better quality on complex, multi-step or highly technical tasks, an appropriate trade when the use case specifically demands it.
Why 200 users doesn't mean 200 concurrent sessions
Two hundred total users rarely means two hundred simultaneous requests; internal tools typically see a fraction of total headcount active at any given moment, often in the range covered for similarly-sized deployments in how many GPUs for 100 concurrent chatbot users. Basing the server configuration on measured or estimated peak concurrency rather than headcount avoids over-provisioning for a load that never actually occurs, which is precisely why the 2x H100 dense configuration above already includes real headroom rather than being sized to a bare-minimum weight footprint.
Two factors that shift the configuration up or down
- Average conversation length and context usage: a deployment where users paste long documents or maintain lengthy multi-turn conversations needs more KV cache headroom than the baseline configuration assumes, potentially pushing the dense 32B deployment from two GPUs toward three for comfortable margin.
- Whether the deployment includes document retrieval (RAG): retrieved context adds meaningfully to per-request token count, as covered in GPUs needed for a RAG system with 500 users, and should be factored into the KV cache budget rather than assumed away.
Redundancy is a secondary benefit of the 2x H100 dense configuration worth calling out explicitly: beyond the extra headroom, splitting the model across two GPUs with tensor parallelism means the deployment does not depend entirely on a single card staying healthy, which a single-GPU configuration for the smaller dense variants would not provide.
Frequently asked questions
Which Qwen 3 variant is right for a 200-user internal assistant?
For general-purpose internal assistant tasks like drafting, summarization and Q&A, the dense 32B variant at FP8 on two H100s typically provides strong quality at meaningfully lower hardware cost than the 235B MoE variant, which is better reserved for complex technical or multi-step reasoning tasks.
Does the MoE 235B variant really need as much memory as a 235B dense model?
Yes, MoE models size to total parameters regardless of active parameter count, as explained in MoE memory vs. active parameters, which is why the 235B variant needs a multi-GPU configuration despite having only about 22B active parameters per token.
Can INT4 quantization reduce the GPU count needed for either variant?
Yes, INT4 roughly halves the FP8 weight footprint for either variant, which could bring the 32B dense model comfortably onto a single GPU or reduce the 235B MoE variant's minimum GPU count, at a small accuracy trade-off worth validating against the target tasks first.
Is a smaller Qwen 3 dense model ever enough for 200 users?
For simpler tasks like classification, extraction or short-form drafting, a smaller dense Qwen 3 variant on a single GPU can be entirely sufficient, and testing against representative real tasks is more reliable than defaulting to the largest available variant.
How Nanobase AI helps
Nanobase AI configures and benchmarks the specific Qwen 3 variant against a customer's expected concurrency before finalizing a server specification, validating that the chosen configuration meets both quality and headroom requirements rather than assuming from parameter count alone.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.