Query rewriting reformulates a user's raw question into a form better suited for retrieval before it is embedded and searched, addressing the common problem that the way people phrase questions rarely matches the wording used in source documents. Simple rewriting techniques include expanding abbreviations, breaking a compound question into sub-questions that are retrieved separately, or using conversation history to resolve pronouns and vague references like that policy into an explicit, searchable phrase. HyDE, short for Hypothetical Document Embeddings, takes a different approach: instead of embedding the question directly, it first asks a language model to write a hypothetical answer to the question, then embeds that generated answer and searches for real documents similar to it, on the theory that an answer-shaped passage matches document-shaped content better than a question-shaped one does. Both techniques add an extra language model call before retrieval, which increases latency and cost per query, so they are most valuable for domains where questions and documents are phrased very differently, such as technical support queries against formal documentation. They are typically evaluated against a baseline of direct embedding to confirm the added cost produces a measurable accuracy gain on the specific corpus. Nanobase AI tests query rewriting and HyDE against direct retrieval before adding either to a production pipeline.
The core problem is vocabulary mismatch, not a missing feature
Users rarely phrase a question using the same words a source document uses to state the answer, and this vocabulary and phrasing mismatch is the single most common reason a technically correct retrieval system misses the right chunk. Query rewriting addresses this mismatch before the query ever reaches the embedding or search step, and several distinct techniques exist under this umbrella, each suited to a different flavor of mismatch rather than one general-purpose fix.
Query rewriting exists because the gap between how people ask and how documents are written is a retrieval problem in itself, independent of chunking or embedding quality.
Technique comparison
| Technique | What it does | Best suited to |
|---|---|---|
| Expansion | Adds synonyms, expands abbreviations, or restates the query in alternate phrasing | Short queries with domain jargon or ambiguous shorthand |
| Decomposition | Splits a compound question into separate retrievable sub-questions | Multi-part questions that reference more than one topic at once |
| Conversational rewriting | Resolves references to prior turns into a self-contained query | Follow-up questions in a multi-turn chat interface |
| HyDE (Hypothetical Document Embeddings) | Generates a hypothetical answer to the question, then embeds that hypothetical answer instead of the raw question | Questions phrased very differently from how the answer is written in source documents |
Each technique targets a distinct symptom, and applying the wrong one for the failure pattern observed adds latency without fixing the actual mismatch.
How HyDE works and where it can hurt rather than help
HyDE asks a language model to write a plausible answer to the user's question without access to any retrieved context, then embeds that generated hypothetical answer and uses it, instead of the raw question, to search the vector store. Because a hypothetical answer is written in a style closer to how a real document would state that answer, it often matches source document phrasing better than the original question does. The risk is that on a topic the model is unfamiliar with or prone to getting wrong, the hypothetical answer can point retrieval toward the wrong part of the corpus entirely, since the search is now guided by the model's guess rather than the user's literal question.
HyDE helps when the model's hypothetical answer resembles real document phrasing, and can actively mislead retrieval when the model's guess is confidently wrong.
A decision approach for which technique to apply
- Diagnose the failure pattern first: is the issue vocabulary mismatch, a compound question, or a conversational follow-up losing context, since each points to a different technique.
- Start with the cheapest applicable fix, expansion or conversational rewriting typically add a single lightweight step, before reaching for a heavier technique like HyDE or decomposition.
- Test HyDE specifically on domain areas where the model has reliable background knowledge, and be cautious applying it broadly across a highly specialized or niche corpus where the model may guess incorrectly.
- Combine decomposition with separate retrieval per sub-question rather than a single merged query, since merging sub-questions back into one search query reintroduces the original compound-question problem.
- Measure retrieval recall with and without the rewriting technique on a real evaluation set before adopting it broadly, since query rewriting adds latency that should be justified by a measurable recall improvement.
Matching the rewriting technique to the diagnosed failure pattern, and validating the improvement, prevents adding latency for a fix that does not address the actual problem.
Frequently asked questions
Does query rewriting always improve retrieval?
No, it helps specifically when vocabulary mismatch, compound questions, or lost conversational context are the actual cause of poor retrieval; applied to a query that already retrieves well, it adds latency without a corresponding benefit.
Can HyDE and hybrid search be used together?
Yes, HyDE's hypothetical answer can be embedded and searched via vector similarity while the original query still runs through BM25 keyword search in parallel, combining the strengths of both in a single hybrid retrieval step.
Is query rewriting expensive in terms of latency?
It adds at least one extra language model call before retrieval begins, which is a real latency cost, though typically small relative to the generation step that follows, and worth it when it measurably improves recall on the queries that were failing.
How do we handle a multi-turn conversation where a follow-up question depends on prior context?
Conversational rewriting resolves the follow-up into a self-contained query by incorporating relevant context from prior turns before retrieval, rather than searching with the follow-up's literal, often ambiguous, wording alone.
How Nanobase AI helps
Nanobase AI diagnoses the specific retrieval failure pattern in a client's system before applying a query rewriting technique, testing HyDE, decomposition, and expansion against real query logs rather than adding rewriting broadly by default. See the related guide on why RAG returns irrelevant results for a broader diagnostic approach and our solutions for the full retrieval stack.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.