How much VRAM do embedding models and rerankers need?

Embedding models and rerankers are far smaller than the generative LLMs they typically support in a RAG pipeline, so their VRAM needs are modest by comparison. Popular open embedding models such as BGE, E5 or GTE variants generally range from under 100 million to about 1 billion parameters, needing roughly 1 to 4 GB of VRAM in FP16 including some overhead for batch processing, while larger multilingual embedding models can reach 4 to 8 GB. Cross-encoder rerankers are similarly compact, usually in the hundreds of millions of parameters, and typically fit in 2 to 6 GB of VRAM depending on batch size and sequence length, since reranking processes query-document pairs together rather than just single texts. Because both model types are small relative to a 7B or larger generation model, they are commonly run on the same GPU as the LLM when there is spare capacity, or on a modest dedicated GPU such as an L4 or RTX 4000 when isolating retrieval components from generation for reliability. Batch size during indexing has a bigger effect on throughput than on peak memory for these smaller models. Nanobase AI, an NVIDIA Inception Program member, sizes the full RAG stack, not just the generation model, when planning GPU capacity.

Why these models are sized differently from the LLM they support

Embedding and reranker models are trained for a narrow task, encoding text into a fixed-length vector or scoring a query-document pair, rather than generating open-ended text, which means they carry none of the KV cache growth that dominates generative LLM sizing. Their memory footprint is essentially fixed regardless of how long a conversation runs, since there is no autoregressive decoding state to accumulate. The sizing question for embedding and reranker models is almost entirely about throughput and batch size, not about memory capacity, which flips the usual GPU sizing priorities for this part of a RAG stack.

Common models and their approximate footprint

These figures already include a reasonable batch-processing overhead beyond bare weights; actual usage shifts within this range depending on maximum sequence length and how many documents are batched per request.

Where to place them relative to the generation model

Because both model types are small compared to a 7B-or-larger generation model, three placement patterns cover nearly every deployment:

1. Co-locate on the same GPU as the LLM when the LLM's precision and quantization leave a few GB of spare VRAM, which is the most cost-efficient option for low-to-moderate query volume.
2. Dedicate a modest GPU (an L4, RTX 4000-class card, or a spare older GPU) when embedding or reranking throughput needs to scale independently of the generation model, such as during a large one-time document ingestion job.
3. Run embedding and reranking as a separate service tier entirely, which matters most when isolating retrieval components for reliability so a generation-side memory spike cannot take down search.

Throughput, not capacity, is usually the real constraint

Because these models process short-to-medium text chunks in large batches rather than long autoregressive sequences, the practical limit teams hit is documents-per-second during bulk indexing, not GPU memory. Doubling batch size on a co-located GPU can meaningfully speed up indexing without materially increasing peak VRAM, since activation memory for these architectures is small relative to a decoder LLM's KV cache. This is the opposite tuning lever from the generation model, where batch size trades directly against KV cache headroom, as covered in how batch size affects GPU memory and throughput.

Budget the RAG generation model first, then treat embedding and reranking capacity as a rounding error on top, adding a dedicated small GPU only when ingestion volume or query latency requirements specifically justify it.

Frequently asked questions

Can I run the embedding model and the LLM on the same GPU?

Yes, in most cases. As long as the LLM's weights and KV cache leave a few gigabytes of headroom, co-locating an embedding model and even a small reranker is common and cost-effective, particularly for internal tools with moderate query volume rather than high-throughput production search.

Do rerankers need more memory than embedding models?

Not significantly more for equivalent parameter counts, though cross-encoder rerankers process query and document together rather than encoding each independently, which can slightly increase activation memory per batch at longer document lengths, generally still within a few gigabytes.

Does a larger embedding model always improve RAG quality enough to justify the extra VRAM?

Larger embedding models often improve retrieval quality, but the gain is workload-dependent; testing retrieval accuracy on representative documents is more reliable than assuming a bigger model is automatically worth its modest extra memory cost.

Should embedding models be quantized like LLMs are?

Quantization is less commonly applied to embedding and reranker models since their memory footprint is already small, and the quality-versus-savings trade-off rarely justifies the added complexity compared to quantizing the much larger generation model instead.

How Nanobase AI helps

Nanobase AI sizes the full RAG stack, generation, embedding and reranking together, rather than treating GPU capacity as a single-model problem. That includes recommending where to co-locate versus separate these components and validating throughput against real document volumes before deployment. Explore how this fits into a broader RAG vs. fine-tuning decision or see it in action in a live demo.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.