How do I serve embedding and reranker models with vLLM or TEI?

vLLM added support for pooling models, which lets it serve embedding models directly through the same server and OpenAI-compatible embeddings endpoint used for generation models, useful when you want one serving stack for both chat and embedding workloads on the same infrastructure. Hugging Face's Text Embeddings Inference, or TEI, is a dedicated engine purpose-built for embedding and reranker models, with highly optimized batching, low latency for the short sequences typical of embedding workloads, and native support for popular reranker architectures like BGE and Jina, which makes it the faster and more memory-efficient choice when embeddings and reranking are your primary workload rather than a secondary one alongside generation. The practical choice is workload-driven: consolidate onto vLLM if you already run it for generation and embedding traffic is modest, or run TEI as a dedicated service if embedding and reranking volume is high, since its specialization typically yields meaningfully better throughput per GPU for that specific task. Both expose standard REST APIs that integrate cleanly with vector database ingestion pipelines and RAG retrieval steps without custom client code. Nanobase AI, a Silicon Valley AI engineering company, selects and tunes the embedding serving layer as part of its retrieval-augmented generation implementations for enterprise customers.

Why embedding workloads behave differently from generation

Generation and embedding serving look similar from the outside, both accept text and return a result over HTTP, but their internal workload shape is close to opposite. Generation is decode-heavy: one token computed at a time, repeated many times, with latency dominated by memory bandwidth. Embedding inference is a single forward pass per input with no autoregressive loop at all, so its bottleneck is almost entirely about how many short sequences you can pack into one batch and how efficiently the engine handles that packing.

This is why an engine tuned around continuous batching for token-by-token generation does not automatically deliver optimal embedding throughput just because it happens to support the model architecture. The scheduling problem is different enough that dedicated embedding engines exist for a reason.

Embedding inference is a batching-and-packing problem, not a decode-latency problem, and the two need different tuning.

Two architectures worth knowing before you pick a tool

Embedding models and rerankers are not the same architecture despite often being grouped together. A bi-encoder embedding model encodes each piece of text independently into a fixed-length vector, which is what lets you precompute embeddings once and compare them cheaply later with a vector database. A cross-encoder reranker, common in architectures like BGE reranker or Jina reranker, instead takes a query and a candidate document together as one input and outputs a relevance score, which is far more accurate for ranking but cannot be precomputed since it needs both texts at inference time.

This distinction matters for capacity planning: reranking traffic scales with query-times-candidates, not just query count, so a reranking step over the top 50 retrieved documents runs 50 forward passes per query, not one.

Rerankers cost roughly candidate-count times more inference than embedding a single query, and capacity planning should reflect that multiplier explicitly.

Feature comparison for the decision

Consolidate onto vLLM if embedding traffic is a modest side workload next to generation on the same infrastructure; run TEI as a dedicated service once embedding and reranking volume becomes the primary load.

Integration into the retrieval pipeline

Both options expose a REST endpoint that plugs directly into standard RAG ingestion and retrieval flows without custom client code: a document pipeline calls the embeddings endpoint at ingestion time to populate a vector database, and the same endpoint gets called again at query time to embed the user's question before similarity search. Where a reranking step follows retrieval, the reranker service takes the query and the top-N retrieved chunks and returns a re-ordered list, typically run only on the initial candidate set rather than the full corpus, since its per-pair cost is much higher than a vector similarity lookup.

Batch size tuning matters more here than for generation, since embedding requests are often short and numerous (a document chunking pipeline might submit thousands of short passages in a burst), and the right batch size balances GPU utilization against acceptable ingestion latency for the pipeline calling it.

A dedicated embedding service is easiest to reason about when it sits behind the same API contract your generation service already uses, so RAG pipelines can call either without custom logic per engine.

Frequently asked questions

Can one GPU serve both an embedding model and a generation model?

Yes, either by running vLLM in pooling mode alongside a separate generation instance, or by co-locating a lightweight TEI process with a generation engine using MIG or careful memory allocation, since embedding models are typically much smaller than generation models and leave meaningful headroom.

Does quantizing an embedding model hurt retrieval quality?

It can, more so than for generation, since embedding vectors are compared numerically and small precision changes shift similarity scores. Validate retrieval quality (recall at k) on your own evaluation set after quantizing rather than assuming generation-model quantization guidance transfers directly.

How often should reranking be used versus vector similarity alone?

Reranking is worth adding when retrieval precision on the first pass is inconsistent, typically for candidate sets in the tens to low hundreds; it adds latency and compute per query, so it is usually applied to a shortlist from vector search rather than the full corpus.

Is TEI harder to operate than vLLM?

Not meaningfully; it exposes a comparable REST interface and container deployment model. The operational tradeoff is running one more service in your stack rather than added complexity within that service itself.

How Nanobase AI helps

Nanobase AI selects and tunes the embedding and reranking layer as part of its retrieval-augmented generation implementations, choosing between a unified vLLM stack and a dedicated TEI deployment based on actual ingestion and query volume rather than defaulting to one option. This work connects directly to our RAG versus fine-tuning guidance and the broader solutions we deliver for production retrieval systems.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.