How do I calculate GPU memory for an LLM?

GPU memory for an LLM comes down to three components: the model weights, the KV cache, and a smaller overhead for activations and the CUDA and framework context. Weight memory equals the parameter count multiplied by bytes per parameter, so 2 bytes for FP16 or BF16, 1 byte for FP8, and about 0.5 bytes for INT4, meaning a 70 billion parameter model needs roughly 140, 70 or 38 GB respectively just for weights. KV cache size depends on the number of layers, the number of key-value attention heads, the head dimension, the precision used, the sequence length and the batch size, and it grows linearly with both context length and the number of concurrent sequences being served. A practical rule of thumb is to take the weight memory figure and add 20 to 50 percent for KV cache and overhead at moderate context lengths, and considerably more for long-context or high-concurrency workloads. Serving engines like vLLM expose memory utilization settings that let this split be observed directly rather than estimated blind. Nanobase AI, part of the NVIDIA Inception Program, builds this calculation against real workload patterns, not generic formulas, before recommending GPU count and model configuration.

The formula, broken into its three inputs

Total GPU memory needed equals weight memory plus KV cache plus a smaller overhead term. Each has its own formula, and working through them separately, rather than relying on a single rule of thumb, is what makes a sizing estimate defensible.

Weight memory = parameter count × bytes per parameter. Use 2 bytes for FP16/BF16, 1 byte for FP8, and about 0.5 bytes for INT4.

KV cache per token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_value. The leading 2 accounts for storing both keys and values. This is then multiplied by sequence length and by the number of concurrent sequences to get total KV cache; the term is broken down in more depth in our guide to KV cache memory usage.

Overhead = CUDA context, framework buffers and activation memory, typically 2 to 6 GB fixed plus a percentage that scales with batch size.

Key takeaway: three separate calculations, not one blended number, is what turns a sizing estimate into something you can defend and re-check.

A worked example: 70B model, 8K context, 10 concurrent users

This total exceeds a single 80 GB H100, which is exactly the kind of finding this exercise is meant to surface before hardware is purchased rather than after: either move to an H200 (141 GB), add a second H100 with tensor parallelism, or reduce the concurrency or context assumptions.

Key takeaway: a worked calculation for your actual expected concurrency, not just the model's weight size, is what tells you whether one GPU is enough.

A quick script to run your own numbers

def gpu_memory_gb(params_b, bytes_per_param, kv_bytes_per_token,
                   context_tokens, concurrent_sessions, overhead_gb=4):
    weights_gb = params_b * 1e9 * bytes_per_param / 1e9
    kv_gb = (kv_bytes_per_token * context_tokens * concurrent_sessions) / 1e9
    return weights_gb + kv_gb + overhead_gb

# 70B model, FP8 weights, FP8 KV cache, 8K context, 10 users
print(gpu_memory_gb(70, 1, 160_000, 8000, 10))

Running this against a few realistic scenarios, low, expected and peak concurrency, gives a range rather than a single point estimate, which is far more useful when deciding how much headroom to build into a purchase.

Key takeaway: a short script that separates weights from KV cache lets you re-run the estimate quickly as concurrency or context assumptions change.

Mistakes that throw the estimate off

The most common error is using active parameters instead of total parameters for a mixture-of-experts model, which understates weight memory by an order of magnitude for models like DeepSeek or Qwen 3. The second is estimating KV cache from an older full multi-head attention formula rather than checking whether the target model uses grouped-query attention, which can overstate KV cache by three times or more. The third is forgetting that overhead scales somewhat with batch size rather than staying fixed, which matters once concurrency climbs into the dozens of simultaneous requests. Re-deriving each term from the model's actual configuration file, rather than reusing a figure from a different model, avoids all three.

Key takeaway: most sizing errors come from reusing a formula input from a different model rather than checking the target model's actual configuration.

Frequently asked questions

Where do I find the num_layers and num_kv_heads values for the KV cache formula?

They are published in the model's configuration file, typically config.json on Hugging Face, under fields like num_hidden_layers and num_key_value_heads. Models using grouped-query attention have fewer key-value heads than attention heads, which is what keeps their KV cache smaller than older architectures.

Why does my actual memory usage differ from the calculation?

Serving engines reserve additional memory for CUDA graphs, activation buffers during the prefill phase, and internal fragmentation, which the basic formula does not capture in full. Treat the formula's output as a floor, and validate against the serving engine's own memory utilization setting.

Does batch size appear in the formula?

Yes, indirectly: KV cache scales with the number of concurrent sequences being processed, which is effectively the serving batch size. Larger batches multiply the KV cache term directly, which is often the fastest way memory runs out under load.

Is there a shortcut for MoE models like DeepSeek or Qwen 3?

Use total parameters, not active parameters, in the weight memory formula, since every expert must be resident in memory. The KV cache formula is unchanged, since KV cache depends on attention layer structure, not on the MoE routing.

How Nanobase AI helps

Nanobase AI, part of the NVIDIA Inception Program, builds this calculation against real workload patterns rather than generic formulas, validating the estimate with actual load tests on candidate hardware before recommending GPU count and model configuration. Our guide to choosing between vLLM, TensorRT-LLM, Ollama and SGLang covers how each engine reports and manages this memory in practice.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.