Sizing GPUs for a coding assistant like Qwen 3 Coder differs from a general chatbot mainly in context length and latency sensitivity, since coding workflows routinely include entire files or multiple files of surrounding code as context, pushing typical context usage well above what a general chat assistant sees, and interactive features like autocomplete demand very low time-to-first-token to feel responsive while a developer is typing. This combination means KV cache usage per active session is often larger than a chat-only deployment of the same model size would suggest, so budgeting extra headroom for context length matters more here than for short-form conversational use cases. For a team-wide deployment, a dense model in the 14B to 32B range in FP8 on one to two H100 or H200 GPUs typically covers a moderate-size engineering team with room for multiple concurrent long-context sessions, while larger coding-specific models may need a multi-GPU configuration similar to any other 70B-class deployment. Prioritizing low latency over maximum batch throughput is usually the right trade-off for this use case, since developers feel delay more acutely during interactive coding than during a chat conversation. Nanobase AI sizes coding-assistant deployments around the specific context length and latency profile developers actually need, not a generic chatbot template.
Two things a chatbot sizing formula misses for code
Coding workflows push a serving deployment in two directions a general chatbot rarely does at the same time: context length and latency sensitivity. A single request for code completion or a refactor commonly includes an entire file, several related files, or a large surrounding code block as context, which is a fundamentally different starting point than a short chat message. At the same time, autocomplete-style features demand very low time-to-first-token to feel responsive while a developer is actively typing, a latency bar tighter than most conversational chat use cases need to meet. Sizing a coding assistant on parameter count alone, without separately budgeting for both elevated context length and tighter latency, is the most common way these deployments underperform after launch.
Context and latency requirements by workload type
| Coding workflow | Typical context per request | Latency sensitivity |
|---|---|---|
| Inline autocomplete | Moderate (surrounding function/file) | Very high; sub-second time-to-first-token expected |
| Chat-style Q&A about code | Low to moderate | Similar to general chat |
| Multi-file refactor or review | High (multiple files) | Moderate; users tolerate more wait for larger tasks |
| Repository-wide search-and-explain | Very high | Lower; often treated as a background task |
This spread means a single coding assistant deployment often needs to serve very different latency and context profiles simultaneously, which argues for prioritizing low latency and adequate KV cache headroom over maximizing raw batch throughput, the opposite tuning priority from a high-volume, latency-tolerant batch workload.
A sizing approach for a team-wide deployment
- Estimate typical context per request from actual developer workflows, not from a generic chat assumption, since even "simple" autocomplete requests often carry more surrounding code than a chat message carries text.
- Budget extra KV cache headroom beyond a same-size chat deployment, following the same per-token math covered in context length's impact on GPU memory, since coding sessions run longer effective context more consistently.
- Choose a model size matched to the team's needs: a dense model in the 14B to 32B range in FP8 on one to two H100 or H200 GPUs typically covers a moderate-size engineering team with room for multiple concurrent long-context sessions.
- Scale up for larger coding-specific models: models built specifically for code generation at 70B-class sizes need a multi-GPU configuration similar to any other 70B deployment, following the guidance in how many GPUs for a 70B model.
- Tune for latency over batch size, since developers feel delay more acutely during interactive coding than during a chat conversation, which may mean accepting a lower maximum concurrent batch size in exchange for consistently fast time-to-first-token.
Why the latency trade-off deserves explicit attention
A serving configuration tuned purely for maximum throughput, large batch sizes, high gpu_memory_utilization for KV cache, can inadvertently increase latency variance under load, which matters more for autocomplete than for a chat assistant a user is prepared to wait a few seconds for. Prioritizing low latency over maximum batch throughput is usually the right trade-off for this use case, even if it means the same GPU serves somewhat fewer peak concurrent sessions than a throughput-optimized configuration would allow.
Frequently asked questions
Does a coding assistant need a bigger model than a general chatbot?
Not necessarily bigger, but the model needs strong code-specific training; a well-trained 14B to 32B coding model often outperforms a larger general-purpose model on coding tasks specifically, making model choice at least as important as raw parameter count.
How many engineers can one H100 support for a coding assistant?
It depends heavily on average context length and how many use autocomplete versus occasional chat-style questions; a moderate-size team's autocomplete traffic, with its long per-request context, consumes KV cache faster per user than an equivalent chat deployment, so testing against real usage patterns is more reliable than a generic per-GPU user count.
Should autocomplete and chat-style code Q&A run on the same GPU?
They can share a GPU if capacity allows, but treating them with different latency priorities in the serving configuration, prioritizing autocomplete requests for low latency, helps avoid one workload degrading the other's user experience.
Is INT4 quantization acceptable for a coding assistant?
It is used in production for coding assistants, though code generation tasks can be more sensitive to precision loss than general text generation for some use cases, making it worth validating output quality on real code samples before committing to an aggressive quantization level.
How Nanobase AI helps
Nanobase AI sizes coding-assistant deployments around the specific context length and latency profile developers actually need, not a generic chatbot template, benchmarking real code-workflow traffic before finalizing a GPU recommendation.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.