![\"A](\"/858ec6ff87bdefd106d04a2e00245dc7/query-transform-pipeline.svg\"){kind=spacer}
An operator on shift types “did it break again?” into the copilot. The retriever embeds that four-word string, searches the index, and hands back the nearest chunks, which could be about anything, because nothing in the query says what “it” is. A minute earlier the same person had asked about a T1_US_FNAL transfer error, so a human would know exactly what they meant. The index has no memory of that turn. It only saw four words with a pronoun in them.
Most RAG (retrieval-augmented generation) debugging skips past this failure. When retrieval returns junk, the reflex is to blame the index: bad chunks, the wrong embedding model, a missing reranker. Sometimes that is right. But often the chunk you need is sitting in the index, embedded perfectly well. The problem is that the query you searched with looks nothing like the document that answers it. You cannot fix that downstream. You fix it by rewriting the query before it reaches the retriever.
\nThis post is for engineers who already have a working RAG pipeline and keep watching it miss on short questions, vague questions, and follow-ups. I will cover why the raw query is so often the weak link, then three techniques that sit in front of retrieval:
\nEach one buys you something different, and each can quietly make retrieval worse if you use it in the wrong place.
\nDense retrieval compares a query vector against document vectors, so it only works when the two land near each other in embedding space. The trouble is that a question and its answer are written in different registers (styles of language). The question is short, informal, and often assumes context the asker already has. The document is a full paragraph of prose that never repeats the question back.
\nTake “why is FNAL slow” and a logbook entry titled “FTS staging backlog on the US Tier-1 causing transfer throughput degradation.” They are about the same thing. Their embeddings can still sit far apart, because almost none of the words overlap and the two texts have different shapes.
\nThat gap shows up in three recurring forms.
\nToo short to be specific. A three-word query carries almost no signal. There is not enough text for the embedding to land anywhere precise, so it lands in a vague neighborhood and pulls back a vague set of chunks.
\nVocabulary mismatch. The user says “slow”; the document says “throughput degradation.” The user says “FNAL”; the document says T1_US_FNAL. Dense retrieval is supposed to bridge synonyms better than keyword search does, and it partly does. That is exactly why the cases it misses are easy to overlook. Hybrid search recovers some of these by adding exact-match BM25 keyword scoring back into the mix, but it cannot invent a term the user never typed.
Conversational follow-ups. In a chat interface, half the questions are not standalone: “did it break again?”, “what about the other site?”, “why?” They make sense only against the previous turns. Embed them literally and you retrieve for the wrong thing entirely.
\nThe fix for all three is a transformation stage between the user and the retriever. The index stays exactly as it is; you change what you ask it.
\n
If you run a chat copilot, build this one first. It fixes the most common failure for the least effort.
\nThe idea is to use the conversation history to rewrite a context-dependent question into a standalone one before retrieval. “did it break again?” becomes “Did the T1_US_FNAL transfer error recur after the fix?” The rewritten query stands on its own, so the retriever now has something to work with.
It takes one LLM call with the history and the latest question. The code below builds that prompt, skips the call when there is no history, and retrieves with the rewritten query:
\nCONDENSE_PROMPT = \"\"\"Given the conversation and a follow-up question,\nrewrite the follow-up as a standalone question that includes the\nentities and context it depends on. Do not answer it. If it is already\nstandalone, return it unchanged.\n\nConversation:\n{history}\n\nFollow-up: {question}\nStandalone question:\"\"\"\n\ndef condense(history: list[dict], question: str) -> str:\n if not history:\n return question # nothing to fold in, skip the call\n convo = \"\\n\".join(f\"{m['role']}: {m['content']}\" for m in history[-6:])\n rewritten = llm.generate(\n CONDENSE_PROMPT.format(history=convo, question=question)\n ).strip()\n return rewritten\n\n# then retrieve with the rewritten query, not the raw one\nstandalone = condense(history, user_question)\nchunks = retriever.search(standalone, k=8)Two details matter:
\nThis is the “condense question” pattern that LangChain’s history-aware retriever formalizes. You can write it in a dozen lines without the framework.
\nContextualizing fixes follow-ups. It does nothing for a standalone question that is simply too vague or unluckily phrased. For that, generate several rewordings and search with all of them.
\nThe reasoning: any single phrasing is one sample from the many ways to ask the question, and it might be a bad sample. Ask an LLM for three or four alternate phrasings, retrieve for each, and merge the results. Where one phrasing misses the right chunk, another catches it, so recall goes up. LangChain’s MultiQueryRetriever packages exactly this, but the mechanics are worth seeing directly. The function below searches with the original question plus the generated variants, then fuses the ranked lists:
EXPAND_PROMPT = \"\"\"Generate 3 alternative phrasings of this question,\neach from a different angle, one per line, no numbering:\n\n{question}\"\"\"\n\ndef multi_query(question: str, k: int = 8) -> list[dict]:\n variants = llm.generate(EXPAND_PROMPT.format(question=question))\n queries = [question] + [q.strip() for q in variants.splitlines() if q.strip()]\n ranked_lists = [retriever.search(q, k=k) for q in queries]\n return reciprocal_rank_fusion(ranked_lists)The part people get wrong is the merge. You now have several ranked lists and need one. Do not just concatenate and dedupe, because that throws away the rank information.
\nUse Reciprocal Rank Fusion (RRF) instead, the same fusion I used to combine keyword and vector results in the hybrid search post. RRF scores each document by summing 1 / (k + rank) across every list it appears in. A chunk that shows up near the top of two different rewrites therefore beats one that ranks first in a single list and nowhere else:
def reciprocal_rank_fusion(ranked_lists, k: int = 60):\n scores, seen = {}, {}\n for lst in ranked_lists:\n for rank, doc in enumerate(lst):\n scores[doc[\"id\"]] = scores.get(doc[\"id\"], 0) + 1 / (k + rank)\n seen[doc[\"id\"]] = doc\n ordered = sorted(scores, key=scores.get, reverse=True)\n return [seen[i] for i in ordered]The cost is real. Three rewrites mean three retrieval round trips, plus the generation that produced them. On a vector index that is usually cheap, but it is not free, and it adds up if you also rerank afterward.
\n![\"Three](\"/e4413882c8dedaa09d17355d79d949ca/three-rewrites.svg\")
Multi-query still searches with questions. HyDE, from Gao et al. at ACL 2023, takes a stranger route. It asks the LLM to write a fake answer to the question, then embeds that fake answer and searches with it.
\nThe reasoning follows directly from the register problem. Your index is full of answer-shaped documents, so an answer-shaped query lands closer to the right neighborhood than the terse question ever could.
\nIt does not matter if the drafted answer is factually wrong. You never show it to the user; you only use its embedding as a better search key. A plausible-but-wrong paragraph about FTS staging backlogs still sits near the real logbook entries about FTS staging backlogs. In code, the only change from normal retrieval is what gets embedded:
\nHYDE_PROMPT = \"\"\"Write a short, plausible paragraph that could answer\nthis question. It does not need to be correct, just realistic in style\nand terminology:\n\n{question}\"\"\"\n\ndef hyde_search(question: str, k: int = 8) -> list[dict]:\n draft = llm.generate(HYDE_PROMPT.format(question=question))\n return retriever.search_by_text(draft, k=k) # embeds the draft, not the queryHyDE earns its keep when the query and the documents are written in genuinely different registers: terse operator shorthand against formal write-ups, or a domain where the user does not know the right vocabulary. In the paper, it was strong precisely where there were no training labels to fine-tune a retriever, which describes most internal corpora. Still, it is the technique I reach for last, because it is the most likely to backfire, as the next section covers.
\nEvery technique here spends an extra LLM generation to improve retrieval, and each has a specific way of going wrong.
\nThe latency is not free, and it lands early. The rewrite happens before retrieval, which happens before generation. You have added a full model call to the front of the request, and the user sees nothing until it finishes. If you stream the answer over SSE, the rewrite is pure dead time before the first token, so measure it as part of time-to-first-token, not as a background cost. Use a small, fast model for the rewrite; it does not need the model that writes the final answer.
\nRewrites drift. A condense or expansion step can hallucinate an entity that was never in the conversation, or quietly change the question’s meaning. If “why is FNAL slow” gets expanded into “why is the CERN Tier-0 slow,” you now retrieve confidently for the wrong site. Log the rewritten query next to the original whenever a request looks off. The bug is often obvious the moment you can see what you actually searched for.
\nHyDE amplifies wrong assumptions. This failure mode mirrors its strength. If the drafted answer is wrong in the wrong direction, its embedding points at the wrong neighborhood, and you retrieve confidently irrelevant chunks. On a question with a specific factual answer that the user already phrased well, HyDE can lose to plain search with the original query. It helps most on vague questions and hurts most on precise ones.
\nOver-expansion dilutes results. More rewrites are not always better. Past three or four, the extra phrasings start repeating each other. Each additional list you fuse in also gives off-target chunks another chance to accumulate RRF score. Recall stops climbing, and precision slips.
\nIt breaks naive caching. If you cache retrieval results keyed on the query string, a rewrite step changes the key on every request. A non-deterministic rewrite can even produce two different keys for the same question. Cache on the rewritten query, or downstream of it, not on the raw input.
\nNone of this is visible unless you measure. Build a small golden set of real queries with their correct chunks, and score recall with and without each technique, exactly as in the retrieval-quality post. Query rewriting is a change to your retriever, and it deserves the same measurement as any other retriever change.
\nIf I were adding this to a pipeline from scratch, I would resist turning on all three at once. The order I would follow:
\nThe order matters because these techniques stack multiplicatively in cost and additively in ways to fail. One well-placed rewrite beats three stacked ones you cannot debug.
\nQuery rewriting decides what your retriever actually gets asked. It is where a copilot stops falling over on “did it break again?” and starts resolving it to the question the operator meant.
\nIt is the query-side complement to the document-side work in the retrieval series behind Archi, the RAG copilot I worked on for CMS computing operations at CERN. Chunking sets the ceiling, hybrid search widens the net, and reranking sharpens the order. Rewriting comes before all of them, because none of that machinery helps if you searched for the wrong thing. Fix the question first, then let the rest of the pipeline do its job.
\nDiagrams by M. Hassan Ahmed, released under CC0. Image credit: original work by the author.
","frontmatter":{"title":"Query Rewriting for Better RAG Retrieval","date":"2026-07-23T00:00:00.000Z","description":"Short, vague, follow-up questions don't match how your docs are written. How query rewriting, multi-query expansion, and HyDE fix retrieval before it runs.","thumbnail":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAAB+0lEQVQoz1WPa2+bMBSG+dCu2GAbjAHbmFsgkAuEpMqaNE2bLU2rrtKmTdq0af//h+ykUj9MenR0dPxeZMttVqQ94zZLp16wyXW1f8q2x/HDi7k5VPvncnfKb4+wFLtH0i6d8QBK0oB+ZTnUODTBrma8ELKN1KQbHtrZbb86NNPtYvVpvrif9XtYJt1OxE0oW8ZLTBIwWg7PSDhiUU2CkoYlTORl2M9sL0VeajPzNhPbMw4HQUVE4fIcXIDleMaTbZB0LGpIWHtRQ8MxRDCeeUHhi9L1Uz+sAt3xdOBm8OXM8TNwOV5qYZYI02fjrak+ZvVG5te+mlJRRqqRuo0lBBVZtRZmycWUyy5Il+AHF2bGQlQLNS1nu7Lfj/p9mC+ZqDBRiGmb6SsiHT/3VM91z5+++evPQdxBP5RjpqFZ21RdJDUu5ihtr9TI5gaLHO7nCFeSqBLtlnd3/o/ffHvy4zlPFi6YibIgW22Ow/e/sy8/h69/utdf2eGVrg9uM+C0wcUEZy2v1tAWxPNA9QF8W88RVYBlu3HQ3RR3z/nuNLp/Mdsj6za022AzRmqEdYUCg2niqbmn+zNyhliC3BgRaSESXznhJRKXWFwg8QEL+/wQo/dpu1Go62K8HE1ugLxe+CKHI7xCc2STN90bsJ8v/4OppEHKRAYTQO+af1+0SGyi03LaAAAAAElFTkSuQmCC","aspectRatio":1.899441340782123,"src":"/static/16e70e892c537421cf56a7d2f5b2a2c6/40a76/hero.png","srcSet":"/static/16e70e892c537421cf56a7d2f5b2a2c6/c972b/hero.png 340w,\n/static/16e70e892c537421cf56a7d2f5b2a2c6/27625/hero.png 680w,\n/static/16e70e892c537421cf56a7d2f5b2a2c6/40a76/hero.png 1360w,\n/static/16e70e892c537421cf56a7d2f5b2a2c6/ed396/hero.png 2000w","sizes":"(max-width: 1360px) 100vw, 1360px"}}}}}},"pageContext":{"slug":"/2026-07-23-query-rewriting-rag-retrieval/","previous":"blog/2026-07-20-llm-as-a-judge-evaluating-outputs/","next":"blog/2026-07-22-tracing-llm-agent-opentelemetry/"}},"staticQueryHashes":["32046230"]}