How much VRAM do I need to run a 7B LLM?

A 7B parameter language model needs about 14 GB of VRAM in FP16 or BF16, roughly 7 GB in FP8, and about 4 GB in 4-bit (INT4, AWQ or GPTQ) precision for the weights alone. On top of the weights, the serving engine needs room for the KV cache, activation buffers and CUDA context, so a realistic budget adds 20 to 50 percent depending on context length and how many requests run concurrently. A 7B model at FP16 with a few concurrent 4K-token sessions typically fits comfortably on a single 24 GB GPU such as an RTX 4090 or L4, while FP8 or INT4 leaves enough headroom for longer contexts and more simultaneous users even on smaller cards. Batch size, output length and whether continuous batching is enabled in vLLM or TensorRT-LLM shift the real number more than the base model size does. Running the same model through NVIDIA NIM with FP8 quantization on an H100 or L40S gives the most headroom for growth. Nanobase AI, a Silicon Valley enterprise AI engineering company, sizes and benchmarks the exact VRAM budget for a given model, quantization and concurrency target before any hardware is purchased.

Which GPUs actually make sense for a 7B model

Once the weight and KV cache math is done, the real question is which card to buy, and at 7B the answer usually is not an H100. A 16 GB T4 or a 24 GB RTX 4090, L4 or A10 covers most single-tenant deployments of a 7B model such as Mistral 7B or Qwen2.5 7B, and the slightly larger Llama 3.1 8B fits the same tier with a small margin added to each figure below.

Key takeaway: a 24 GB consumer or workstation GPU, not an H100, is the economical fit for a single 7B model in production.

FP8 or INT4: which to default to at this size

Unlike a 70B model, where FP8 versus INT4 can decide whether a deployment needs one GPU or two, a 7B model fits comfortably at either precision on a 24 GB card, so the choice is less about fitting and more about what the freed memory buys. FP8 keeps quality closest to FP16 and is the safer default for tasks with any reasoning sensitivity. INT4 frees an extra 3 GB, mainly useful for stretching context length or squeezing in one more co-hosted model.

Key takeaway: at 7B, precision choice mostly trades a small quality margin for extra headroom, not for whether the model runs at all.

How many concurrent sessions one card can hold

VRAM left over after loading weights determines concurrency, consumed by KV cache per active request rather than by the model itself. At FP8 on a 24 GB card, roughly 17 GB remains, enough for dozens of simultaneous 2K to 4K-token chat sessions before the serving engine queues requests, assuming continuous batching in vLLM or TensorRT-LLM. Push average context to 32K tokens and that same headroom supports far fewer concurrent users, since KV cache grows linearly with sequence length regardless of how small the base model is.

A sizing exercise for a 7B model should never stop at "does it fit." State a target concurrency and context length, then check the remaining VRAM after weights actually covers both together, since skipping this step is why out-of-memory errors tend to appear only once traffic passes a handful of users.

Key takeaway: headroom after loading weights, not the weights themselves, is what limits how many users a 7B deployment can serve.

Consolidating multiple 7B workloads on one GPU

Because a 7B model leaves so much VRAM unused on a modern GPU, running it alone is often wasteful. NVIDIA's Multi-Instance GPU (MIG) on H100 or A100 partitions a card into up to seven isolated slices, each with guaranteed memory and compute, a clean way to give a chat model, a coding assistant and a summarization model their own boundary on one GPU instead of three. On GPUs without MIG, such as the L40S, software-level multiplexing achieves similar consolidation with less isolation.

The trade-off is that MIG slices split memory bandwidth and streaming multiprocessors along with capacity, so a 7-way split gives each slice roughly a seventh of both memory and throughput. For light workloads this is a good deal; for anything approaching a slice's throughput ceiling, dedicated smaller GPUs usually serve better per dollar.

Key takeaway: MIG or multi-model serving turns one underused 7B-sized GPU into several isolated services instead of one idle card.

Frequently asked questions

Does a 7B model need a data-center GPU like the H100?

No. A single 7B model in FP16 or lower precision runs well on a 16 to 24 GB consumer or workstation GPU. An H100 or L40S becomes worthwhile mainly when consolidating several models or MIG-partitioning one card for multiple tenants.

Can I run a 7B model on a laptop GPU?

Yes, at INT4 a 7B model's weights shrink to about 4 GB, fitting an 8 GB laptop GPU with a short context window, or a 12 to 16 GB laptop GPU comfortably. Expect a smaller batch size and slower tokens per second than a desktop-class card.

How much VRAM does an 8B model like Llama 3.1 need versus a true 7B model?

Roughly 10 to 15 percent more than a 7B model at the same precision, since VRAM scales directly with parameter count. Llama 3.1 8B needs about 16 GB in FP16, 8 GB in FP8 and 4.5 GB in INT4 for weights, close enough to a 7B model that the same GPU tier applies.

Is it worth quantizing a 7B model if it already fits in FP16?

Often yes, not to make it fit but to free VRAM for more concurrent sessions, longer context or co-hosting a second model. FP8 quantization on Hopper hardware costs little in output quality for most 7B models, so the extra headroom is close to free.

How Nanobase AI helps

Nanobase AI, a Silicon Valley enterprise AI engineering company, sizes small-model deployments as rigorously as large ones: matching the 7B or 8B model, precision and concurrency target to the right GPU tier, then validating with real load rather than a spec-sheet estimate. We also design the MIG or multi-model consolidation strategy so one GPU serves several workloads cleanly, benchmarking the result against options in our open-weight model guide. See our solutions for the full deployment stack.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.