![\"Hybrid](\"/8be0ae2ebc57fee6af18ca3485348132/hybrid-retrieval.svg\")
Someone asks the copilot “what fixed the T1_US_FNAL transfer error last spring?” The retrieval step returns three paragraphs about transfer errors in general and nothing about T1_US_FNAL. The embedding model saw a string that looks like every other site name and put it in roughly the same region of vector space as the rest. The one token that mattered got averaged away.
This failure is what pushes most RAG (retrieval-augmented generation) systems from pure vector search to hybrid search: running keyword search and vector search side by side and merging the results. Dense retrieval is good at meaning and bad at exact strings. Keyword search is the reverse. If your corpus is full of identifiers, error codes, config keys, or proper nouns (and internal operations docs always are), you want both, plus a principled way to merge them.
\nThis post is for engineers who already have a working vector RAG pipeline and keep hitting queries it retrieves badly. It covers what each retriever is actually good at, how to fuse two ranked lists without hand-tuning weights, why a reranker belongs at the end, and where the whole thing still breaks.
\nBM25 is the lexical (keyword-matching) baseline, and it has held up for thirty years. It scores a document by how often the query terms appear in it, with two adjustments:
\nBM25 runs over an inverted index (a map from each token to the documents that contain it), so it is fast and matches tokens exactly. Search for T1_US_FNAL and BM25 finds the documents that literally contain T1_US_FNAL. Robertson and Zaragoza give the full statistical account in The Probabilistic Relevance Framework: BM25 and Beyond.
Dense retrieval works differently. You embed each chunk into a vector with a model like Sentence-Transformers, embed the query the same way, and return the chunks whose vectors sit closest to the query’s. Closeness in that space means similar meaning. A query about “moving data between sites” can match a chunk that says “transfers between grid tiers” with no shared words.
\nThat is the whole appeal, and it is also the weakness. The model is trained to collapse surface differences, so it collapses the differences you care about too. T1_US_FNAL and T2_DE_DESY are near-identical to an embedding model and completely different to an operator.
Neither retriever is wrong. They are good at different queries, and real questions are a mix. So run both and combine them.
\nThe naive version is to run both searches, add the scores, and sort. It does not work, because the scores are not comparable. BM25 produces an unbounded positive number that depends on term statistics. Cosine similarity lives in a fixed range near zero to one. Add them, and BM25 either dominates or vanishes depending on the query. You end up hand-tuning a weight per corpus, and it drifts the moment the data changes.
\nThere are two ways out. Different engines default to different ones, so it is worth knowing both.
\nScore normalization rescales each retriever’s scores onto a common range, then combines them. OpenSearch’s normalization processor does this inside a hybrid query. It supports min-max and L2 normalization, then combines the results with an arithmetic, geometric, or harmonic mean. It works, but you still choose a normalization method and a combination weight. Min-max in particular is sensitive to a single outlier score stretching the range.
\nRank fusion throws the scores away entirely and combines the lists by rank, meaning each document’s position in each list. That is what the next section covers.
\nReciprocal Rank Fusion (RRF) is the method that quietly ended up inside most hybrid search implementations, including Elasticsearch and newer OpenSearch. The formula fits on one line: a document’s fused score is the sum of 1 / (k + rank) over every ranked list it appears in.
The function below implements it. Each input is one retriever’s output, ordered best first.
\ndef reciprocal_rank_fusion(ranked_lists, k=60):\n \"\"\"ranked_lists: a list of retriever outputs, each an ordered list of doc ids.\"\"\"\n scores = {}\n for ranking in ranked_lists:\n for rank, doc_id in enumerate(ranking, start=1):\n scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)\n return sorted(scores, key=scores.get, reverse=True)\n\nfused = reciprocal_rank_fusion([bm25_hits, vector_hits])\ntop_k = fused[:8]That is the whole algorithm. Notice what is missing: no scores from either retriever, no normalization, no per-corpus weight. A document only needs its position in each list. For example, a chunk ranked first by BM25 and eighth by vector search scores 1/61 + 1/68.
A chunk that both retrievers rank near the top climbs to the top of the fused list, which is exactly the behavior you want. Agreement between two independent methods is a strong signal.
\nThe k = 60 is not arbitrary. In the original 2009 paper by Cormack, Clarke, and Grossman, 60 was the constant that scored best across their benchmarks, and it has been the default everywhere since. Its job is to flatten the curve so the gap between rank 1 and rank 2 is not enormous. A larger k makes the fusion more forgiving of exact position. A smaller one rewards the very top ranks more heavily.
I have never needed to move it. The paper’s headline result was a parameter-free formula beating trained learning-to-rank methods, which is a good reminder that the simple thing is often enough.
\nThe diagram below shows the pipeline end to end.
\n
The code above is the right way to understand RRF. It is not always the right way to ship it. Pulling a few hundred candidates from each retriever across the network just to fuse them in your app adds a round trip and moves ranking logic away from the data. If your search engine already does hybrid queries, let it.
\nIn OpenSearch, you register a search pipeline once. After that, a single hybrid query carries both a lexical clause and a k-NN (nearest-neighbor vector) clause. The pipeline normalizes and combines the two result sets server-side and hands you one ranked list. Elasticsearch exposes RRF directly as a retriever.
The Python version still earns its place in two cases:
\nFusion gives you a good candidate set, not a final order. RRF knows only about ranks. It has never looked at whether a chunk actually answers the question.
\nA cross-encoder reranker does look. It is a model that takes the query and one candidate chunk together and scores their relevance directly. Vector search, by contrast, compares two vectors that were embedded separately and never saw each other.
\nThe code below scores the top 30 fused candidates with a cross-encoder, re-sorts them, and keeps the best 5 as context.
\nfrom sentence_transformers import CrossEncoder\n\nreranker = CrossEncoder(\"cross-encoder/ms-marco-MiniLM-L-6-v2\")\n\npairs = [(query, doc_text[doc_id]) for doc_id in fused[:30]]\nrerank_scores = reranker.predict(pairs)\n\nreranked = [doc for _, doc in sorted(zip(rerank_scores, fused[:30]), reverse=True)]\nfinal_context = reranked[:5]The pattern is retrieve wide, rerank narrow. Fusion hands you thirty candidates cheaply. The cross-encoder does the expensive, careful read on just those thirty, and only the top five reach the model’s context.
\nIt costs one extra model call. It is usually the single biggest quality jump after adding lexical search, because it is the first stage that reads the query and the chunk as one thing.
\nTokenization mismatch on the lexical side. BM25 only matches what your analyzer (the step that splits text into tokens) produced. If the indexer splits T1_US_FNAL into t1, us, fnal but the query analyzer keeps it whole, they never meet. Your keyword branch then silently returns nothing for the exact query it was supposed to save. Index-time and query-time analysis have to agree, and that deserves an explicit test.
Fusing lists of wildly different lengths. If BM25 returns four hits and vector search returns two hundred, RRF still works, but the fused ranking is dominated by whichever list is longer in its tail. Cap both retrievers at the same depth before fusing. The top 50 to 100 from each is plenty, since anything past that rarely survives reranking anyway.
\nBlaming retrieval when the problem is chunking. Hybrid search cannot retrieve a fact that got split across two chunks so that neither holds the whole answer. When a query fails, check that the answer exists intact in a single chunk before you tune fusion weights. This sits upstream of everything here, and it is where I have wasted the most time.
\nKeeping that index fresh is its own problem. I wrote about detecting changed files and re-embedding only what moved in incremental indexing for RAG pipelines.
\nLatency creep. Two retrievers plus a reranker is more work than one vector lookup. The retrievers run in parallel, so that part is close to free, but the cross-encoder works through the candidates sequentially. If you stream answers to a user, budget for it, and try reranking fewer candidates before you consider dropping the step.
\nI reached for a reranker too late. On early RAG systems, my instinct was to tune the retrievers: better embeddings, a bigger candidate pool, fiddling with weights. The cheapest large gain was bolting a cross-encoder onto the end and letting the retrievers stay rough. Retrieve wide and rerank narrow is a better default than retrieve precise, and it is less work.
\nI also used to normalize and weight scores by hand, one setting per corpus, and then relearn the setting every time the data shifted. RRF made that whole category of tuning disappear. Ranks are stable in a way raw scores are not, and giving up the scores cost me nothing I could measure.
\nHybrid retrieval is the search layer under Archi, the RAG copilot I worked on for CMS computing operations at CERN. Lexical search catches the site names, error codes, and JIRA ticket IDs that operators actually type. Vector search handles the “how did we handle something like this before” questions that never share vocabulary with the docs.
\nThe same two-retriever pattern showed up in ARGRAG, the enterprise-document RAG system my teammate and I built for the Argusa AI Challenge. There the corpus was a pile of PDFs, spreadsheets, and source files, and no single retrieval method covered all of it.
\nOne retriever is a bet that your queries will look like your training data. Hybrid search is what you do when you know they won’t.
\nImage credit: hybrid search diagrams by M. Hassan Ahmed, created for this post, released under CC0 (public domain).
","frontmatter":{"title":"Hybrid Search for RAG: BM25 + Vectors","date":"2026-07-02T00:00:00.000Z","description":"Vector search alone misses exact IDs and error codes. Here's how to combine BM25 keyword search with dense retrieval, fuse the rankings with RRF, and rerank.","thumbnail":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAAB40lEQVQoz02PWY/aMBRG89LJQpwEKAnxTUycfSGELANMYKCqAKmsbacPfen//xs1g0pHOrqyfb/ja3MmrAy8pP52Wv8sqh/Z5DwuLvn0ytZ5cWHbsvk1Kb/fDosry2STS9m8jehXZnGSBoJqagalcZlM22S6TIo2mryk5Wpcr2+1emXnYb7I6/Wk2bhJ7caVplNmcZ0u6XRHkmbzCD8hS1DJE7J5xUY9Imnkk4xZS+5SQYEnhHkF2IJHwFrM4jqa/Q/SN5zuwAGgaeAF1PFGxHML21oQ+8WlSwNnPd3TPrsym/eucJJqMTqqJSDAtpdGUVsmdR6NwMqns/bIvroGs0njXTE++cGybwSiAneLyfBAQDiP/WkaWhgC6j/vzvX5NNvtXWdumlUW79N4z8vGI89JCr4jIqz2YFHE1TikFo6CbHa6bv78nh8PYbCwYZ7F2zTaqX0qouFd4UTFvCMgE2k49UehSxwLIj9ujufi7dwcvvneDPCz6yw957WjgqgM7wrHrnnAy8Pbg4nlAKaAy/ZLfTpU7cYyazBrAgtdH/PyQETmPc9k44GADEkxqG2z2Q5xAi+erbaetwBoiD0HXL07//OcIOsf4TsDSbU1PejjhNXuIOgN494w7BoB+yQb+zH8F4mGUab5x2CsAAAAAElFTkSuQmCC","aspectRatio":1.899441340782123,"src":"/static/dd2bcaa9601c27b33c0e7510855e8462/40a76/hero.png","srcSet":"/static/dd2bcaa9601c27b33c0e7510855e8462/c972b/hero.png 340w,\n/static/dd2bcaa9601c27b33c0e7510855e8462/27625/hero.png 680w,\n/static/dd2bcaa9601c27b33c0e7510855e8462/40a76/hero.png 1360w,\n/static/dd2bcaa9601c27b33c0e7510855e8462/ed396/hero.png 2000w","sizes":"(max-width: 1360px) 100vw, 1360px"}}}}}},"pageContext":{"slug":"/2026-07-02-hybrid-search-rag-bm25-vectors/","previous":"blog/2026-07-03-argocd-sync-waves-ordering-rollout/","next":"blog/2026-07-01-opensearch-workflow-monitoring/"}},"staticQueryHashes":["32046230"]}