Can I run a 32B model on a 24GB GPU?

A 32 billion parameter model generally cannot run on a 24 GB GPU at FP16, since the weights alone need about 64 GB, but it fits with 4-bit quantization, where the weights shrink to roughly 16 to 18 GB, leaving a modest 6 to 8 GB for KV cache, activations and CUDA overhead. That headroom limits the deployment to a fairly short context window and only one or two concurrent requests before running out of memory, so a 24 GB card such as an RTX 4090 or L4 is realistic for a single-user or light-duty deployment of a quantized 32B model like Qwen 32B, not for serving many simultaneous users. AWQ and GPTQ are the two 4-bit formats most commonly used for this scenario, and both integrate with vLLM for reasonably efficient serving despite the tight memory budget. Dropping context length or reducing batch size further is often necessary if out-of-memory errors appear in production. Moving to a 48 GB GPU or an RTX PRO 6000 with 96 GB removes most of these constraints. Nanobase AI, an enterprise AI engineering company based in Silicon Valley, tests the actual context and concurrency limits of quantized models on a given GPU before it goes into production.

What context length is actually usable

Fitting a quantized 32B model on a 24 GB card is only step one; the more useful question is how much of that remaining 6 to 8 GB of headroom translates into usable context and concurrency, since that is what determines whether the deployment is viable for real traffic.

Key takeaway: a 24 GB GPU running a quantized 32B model handles short-context, low-concurrency use well, but long context and multiple simultaneous users compete for the same thin margin.

Configuration settings that matter more than the GPU spec

Two serving-engine settings determine whether a 32B-on-24GB deployment stays stable under load: gpu-memory-utilization and max-model-len in vLLM, or their equivalents in other engines. Setting memory utilization too high leaves no margin for activation spikes during generation, while setting max-model-len above what the actual headroom supports causes failures the moment a request approaches that limit rather than at model load time, which makes the failure mode confusing in production.

vllm serve Qwen/Qwen2.5-32B-Instruct-AWQ \
  --quantization awq \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8

Capping max-model-len conservatively and quantizing the KV cache to FP8 alongside INT4 weights is what makes a 32B model on a 24 GB card practical for more than a single toy request at a time.

Key takeaway: the serving engine's memory and context settings, not just the GPU's raw VRAM, determine whether the deployment survives real traffic.

When to move up instead of squeezing further

A 24 GB card running a 4-bit 32B model is a genuinely useful single-user or light-duty configuration, but it is not the right target for anything resembling production concurrency. Once the requirement grows past one or two simultaneous long-context sessions, the honest options are a 48 GB GPU such as the L40S, which roughly doubles the KV cache headroom without changing quantization, or an RTX PRO 6000 at 96 GB, which comfortably fits the 32B model even at FP16 with substantial room left over. Continuing to push context length or concurrency on the same 24 GB card past this point usually produces diminishing, unstable results rather than genuine capacity.

Key takeaway: treat a 24 GB card running a quantized 32B model as a single-user or evaluation setup, and plan the upgrade path before concurrency demands force it.

Which "32B-class" model you pick changes the math slightly

Not every model in this size band has the identical footprint. Qwen2.5 32B and Gemma 3 27B differ by roughly 15 percent in raw parameter count, which shifts INT4 weight memory by a comparable margin, a small but real difference when the entire remaining budget is only 6 to 8 GB. Gemma 3 27B's slightly smaller size can be the difference between a usable 8K context and a cramped 4K one on the same 24 GB card, so when a model choice is not fixed, comparing the actual parameter counts of candidate models is worth doing before assuming any "32B-class" model behaves identically on constrained hardware.

Key takeaway: a few billion parameters of difference between similarly sized models can meaningfully change the usable context on a tightly constrained 24 GB card.

Frequently asked questions

Can I run Qwen 32B at FP16 on a 24GB GPU?

No. FP16 weights for a 32B model need about 64 GB, far more than a 24 GB card provides. Only 4-bit quantization (AWQ or GPTQ), bringing weights to roughly 16 to 18 GB, makes a 32B model fit on 24 GB at all.

What happens if I exceed the context limit on a memory-constrained setup?

The request typically fails with an out-of-memory error, or the serving engine rejects it before processing if a hard max-model-len is configured. Without that safeguard, a long request can crash the serving process or degrade every other concurrent session.

Is AWQ or GPTQ better for a tight 24GB budget?

Both produce a similar memory footprint and accuracy for most models, so the practical difference usually comes down to which format has better kernel support in your chosen serving engine. AWQ integrates well with vLLM's memory-efficient serving path, which is worth checking against your specific stack.

Should I quantize the KV cache too, or just the weights?

For a tight VRAM budget like 24 GB, quantizing both is usually worthwhile. FP8 KV cache roughly halves that portion of memory versus FP16, directly increasing the usable context length or concurrency without touching model weights again.

How Nanobase AI helps

Nanobase AI, an enterprise AI engineering company based in Silicon Valley, tests the actual context length and concurrency ceiling of a quantized model on a given GPU before it reaches production, tuning the serving engine's memory settings rather than guessing at a safe default. When a 24 GB card genuinely is not enough, we help size the right upgrade path across vLLM, TensorRT-LLM and other serving stacks. See our solutions page for the full engagement model.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.