A reranker is a second-stage model, typically a cross-encoder, that takes the top candidates returned by an initial vector or hybrid search and re-scores them by jointly reading the query and each candidate chunk together, rather than comparing precomputed embeddings independently. This joint scoring is more accurate than a simple similarity comparison because the model can weigh subtle relevance signals it cannot capture when the query and document are embedded separately, and in practice reranking a top-50 or top-100 candidate list down to the top 5 to 10 chunks measurably improves precision and reduces the number of irrelevant chunks that reach the language model. The tradeoff is latency: cross-encoder reranking adds inference time proportional to the number of candidates scored, so most systems retrieve a wider initial candidate set cheaply with vector or hybrid search, then apply the reranker only to that shortlist. Open models such as BGE reranker and Cohere Rerank, alongside NVIDIA NeMo Retriever reranking models, are commonly used in production and can run on a single GPU with modest latency for typical enterprise query volumes. Reranking is one of the highest-return changes a team can make when retrieval quality plateaus. Nanobase AI, a Silicon Valley AI engineering firm, adds reranking to RAG pipelines whenever retrieval precision is the bottleneck.
Reranking is a trade, not a free upgrade
A reranker's accuracy improvement is well established, but it is not a free addition to a pipeline. It requires running a second model, typically a cross-encoder, over every candidate from the first-stage retrieval, which adds both latency and compute cost proportional to how many candidates are reranked. Deciding whether to add one is a trade-off question specific to a system's latency budget and traffic volume, not a default that should be added to every RAG pipeline without measuring the cost.
Reranking trades measurable latency and compute for measurable accuracy, and that trade is worth evaluating explicitly, not assumed.
Where the latency actually goes
| Stage | Typical latency contribution |
|---|---|
| First-stage retrieval (vector or hybrid) | Fast, milliseconds to tens of milliseconds at reasonable scale |
| Reranking 20 to 50 candidates | Adds tens to low hundreds of milliseconds depending on model size and hardware |
| Language model generation | Often the largest single contributor, particularly for longer answers |
Reranking is rarely the dominant cost in total request latency; generation usually is, which changes how much reranking's added cost actually matters.
Self-hosted vs API rerankers
Self-hosted open cross-encoder rerankers, such as models in the BGE reranker or Jina reranker families, run on a GPU alongside other inference workloads and avoid a per-call external dependency, which matters for on-premise or data-residency-constrained deployments. Hosted reranking APIs remove the operational burden of running the model but add network latency and a per-call cost that scales with query volume. The choice mirrors the broader on-premise-versus-API decision for the rest of the LLM stack, and for a fully on-premise RAG deployment, self-hosting the reranker is usually the only option that satisfies the constraint anyway.
The self-hosted versus API decision for reranking usually follows the same data-residency and volume logic as the rest of the inference stack.
When reranking earns its cost and when it does not
Reranking earns its cost clearly when the first-stage retrieval returns a reasonable candidate set but struggles to put the single best chunk at the very top, which is common with hybrid search returning a broad top-20 or top-50. It earns its cost less clearly when the corpus is small enough that first-stage retrieval already ranks well, or when the system's top-k passed to the language model is generous enough that a mediocre first-stage ranking rarely excludes the right chunk anyway. Measuring answer accuracy with and without reranking on a real evaluation set is the reliable way to settle this for a specific deployment rather than assuming it always helps.
Reranking helps most when the first-stage ranking is good enough to include the right chunk but not good enough to rank it first.
Frequently asked questions
How many candidates should be passed into a reranker?
A common range is retrieving 20 to 50 candidates from the first stage and reranking down to the top 3 to 10 passed to the language model, though the right numbers depend on corpus size and latency budget and should be tuned against real queries.
Does reranking reduce hallucinations?
Indirectly, by improving the odds that the most relevant chunk is actually in the context the model sees. It does not directly prevent a model from misusing correct context, which is a separate concern handled through prompting and answer verification.
Can reranking run on CPU instead of GPU?
Small cross-encoder rerankers can run on CPU, but latency increases substantially compared to GPU inference, which matters for interactive applications where response time is user-facing.
Is reranking necessary if we already use hybrid search?
Not strictly necessary, but the two are complementary rather than redundant. Hybrid search improves what gets into the candidate set; reranking improves the order of that candidate set, and combining both consistently outperforms either alone in most evaluations.
How Nanobase AI helps
Nanobase AI, a Silicon Valley enterprise AI engineering company, benchmarks reranking's actual latency and accuracy impact on a client's own corpus before adding it to a production pipeline, and deploys self-hosted rerankers on-premise where data residency requires it. See the related guide on hybrid search fusion and our solutions for the full retrieval stack.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.