What is the GPU memory overhead of CUDA and vLLM beyond model weights?

Beyond the model weights themselves, CUDA reserves a base memory footprint per process, typically a few hundred megabytes to around 1 to 2 GB depending on the driver version and libraries loaded, purely for the runtime context, kernels and buffers needed before any model is even loaded. Serving engines like vLLM add their own overhead on top of that: they reserve a configurable fraction of remaining GPU memory, often defaulting to around 90 percent of available capacity, for the KV cache pool, plus additional memory for activation buffers, CUDA graphs used to reduce kernel launch overhead, and internal bookkeeping structures. Altogether, this non-weight overhead commonly adds up to roughly 5 to 15 percent beyond the raw weight size of a model, which matters when sizing a GPU close to its capacity limit, since underestimating it is a common cause of unexpected out-of-memory errors in production. Setting the memory utilization parameter conservatively during initial deployment and monitoring actual usage is safer than assuming the full advertised VRAM is available for weights and KV cache alone. Nanobase AI, headquartered in Silicon Valley, accounts for this overhead explicitly rather than sizing GPUs against theoretical maximum capacity.

Why "80 GB of VRAM" never means 80 GB for the model

Every number quoted for a GPU's memory capacity describes total physical HBM, not the amount available for a model's weights and serving state. Before any model loads, the CUDA driver and runtime reserve a base context, and once a serving engine starts, it adds its own reservations for the KV cache pool, activation buffers, and internal bookkeeping. Treating the advertised VRAM figure as the usable budget is one of the most common causes of out-of-memory errors that appear only after a deployment has been running for a while, since the gap only becomes visible once concurrency or context length pushes usage close to the true ceiling.

A worked budget example on an H100

This example is deliberately tight to illustrate the point: a 70B model at FP8 on a single H100 looks like it "fits" by weight size alone, but the realistic remaining KV cache budget is a small fraction of the 80 GB headline figure, which directly caps concurrency and context length, a scenario covered in more depth in running Llama 3.3 70B on a single H100.

Where each piece of overhead actually comes from

1. CUDA runtime context is a near-fixed cost per process regardless of model size, driven mainly by driver version and which CUDA libraries (cuBLAS, cuDNN, NCCL) get loaded.
2. Serving engine memory reservation is configurable: vLLM's gpu_memory_utilization parameter sets what fraction of total device memory the engine is allowed to claim, commonly defaulting to around 90 percent, with the rest deliberately left unclaimed as a safety margin.
3. KV cache pool pre-allocation happens once at startup based on the configured utilization fraction minus the model's weight size, which is why changing this single parameter is often the fastest way to diagnose whether an OOM error is a true capacity problem or a configuration one.
4. CUDA graphs, used by modern serving engines to reduce per-step kernel launch overhead and improve latency, trade a modest fixed memory cost for meaningfully better throughput, and can usually be disabled temporarily to free memory during debugging.

Practical guidance for sizing against this overhead

# vLLM example: reserve 85% of GPU memory for weights + KV cache,
# leaving explicit headroom instead of assuming 100% is usable
vllm serve <model> --gpu-memory-utilization 0.85

Setting this parameter conservatively during initial deployment, then raising it gradually while monitoring actual memory usage under real traffic, is safer than assuming the full advertised capacity is available for weights and KV cache combined. Roughly 5 to 15 percent of a GPU's advertised memory should be assumed unavailable for the model itself, and that fraction matters most precisely in the tight-fit scenarios, like a large model on a single GPU, where every gigabyte of KV cache headroom counts.

Frequently asked questions

Does this overhead scale with model size?

The CUDA context portion stays roughly fixed regardless of model size, while the serving engine's activation buffers and CUDA graph memory scale modestly with model architecture (layer count, hidden dimension), so the overhead is proportionally larger for smaller models and proportionally smaller for larger ones.

Can I reduce this overhead to fit a slightly larger model?

Partially. Disabling CUDA graphs, lowering gpu_memory_utilization on the KV cache side is the wrong direction, but tuning batch size limits and disabling optional features can reclaim a modest amount, generally not enough to fit a meaningfully larger model that does not otherwise have headroom.

Why did my deployment work in testing but fail with an OOM error in production?

Testing with light or synthetic traffic often uses far less KV cache than real concurrent production traffic with longer, more varied context lengths, so a configuration that looked fine with headroom in testing can hit the true memory ceiling once realistic load and this overhead are both present.

Is this overhead different between vLLM, TensorRT-LLM and NVIDIA NIM?

The general categories, CUDA context, activation buffers, KV cache pre-allocation, apply across all three, though the exact defaults and tuning parameters differ by engine, which is why validating memory behavior on the specific serving stack being deployed matters more than a single universal number.

How Nanobase AI helps

Nanobase AI, headquartered in Silicon Valley, accounts for this overhead explicitly rather than sizing GPUs against theoretical maximum capacity, tuning serving engine parameters and validating real memory headroom before a deployment goes live. See how this fits into broader Kubernetes GPU Operator and Slurm infrastructure decisions.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.