![\"Two](\"/2aea86b1984b15c0bcabe043c2627b69/bi-vs-cross-encoder.svg\")
An operator asks the copilot “what fixed the T1_US_FNAL transfer error last spring?” The chunk that answers it is in the index, and retrieval finds it. But it comes back at rank 8, and the prompt only carries the top 4 chunks into the model. The answer was retrieved, then thrown away before the model ever saw it.
Reranking closes this gap. Your first-stage retriever is tuned to be fast and to not miss things, so it casts a wide net and orders the catch only roughly. A reranker takes that shortlist and puts the actually relevant chunk on top, so it survives the cut into the prompt.
\nThis post is for engineers who already have retrieval working, hybrid or not, and keep seeing the right document come back somewhere in the top 20 but not the top 3. It covers why the retrieval score is coarse, what a cross-encoder does differently, working code for both a local model and a hosted API, and the failure modes that show up once reranking is in the pipeline.
\nDense retrieval runs on a bi-encoder, a model that encodes the query and each document separately. It embeds every document into a vector ahead of time, embeds the query at search time, and ranks documents by cosine similarity. The reason this scales is exactly the reason it loses precision: the query and the document never meet. Each is compressed into a fixed-length vector on its own, and at query time the model only compares two points in that space.
\nThat compression throws away information ranking needs. The document vector has to summarize the whole chunk before it knows what you are going to ask. Take a chunk that mentions T1_US_FNAL once, in passing, alongside four other sites. Its vector ends up mostly about “transfer errors in general,” and the one token you cared about is averaged into the background.
I wrote about the keyword side of this failure in the post on hybrid search for RAG. Adding BM25 keyword search recovers the exact-match cases, but even then the fusion step is combining two coarse scores. Neither retriever ever reads the query and the chunk side by side and asks: does this specific passage answer this specific question?
\nA cross-encoder asks exactly that. Instead of embedding the two texts separately, it concatenates them as [query] [SEP] [document] and runs the pair through a transformer in one pass. Every token in the query can attend to (directly weigh itself against) every token in the document. The output is a single number: how relevant this passage is to this query.
The idea comes from Nogueira and Cho’s 2019 “Passage Re-ranking with BERT”. They applied it to the MS MARCO passage ranking task and beat the previous best by a wide margin, doing nothing more exotic than this. The diagram below contrasts the two approaches.
\n
The accuracy comes at a cost. Because the score depends on the pair, there is nothing to precompute: you cannot embed your corpus once and reuse it. Every query-document pair is a fresh forward pass through the model. Scoring a million documents this way at query time is a non-starter, which is why you never use a cross-encoder as your retriever. You use it as a second stage, on a shortlist the first stage has already narrowed down.
\nThe pattern is to retrieve wide and cheap, then rerank narrow and precise:
\nFifty forward passes is a latency you can pay. A million is not.
\nThe property to hold onto is that stage one alone sets the recall ceiling. The reranker can only reorder what it is given. If the answer is not among the 50 candidates, no amount of reranking invents it.
\nSo the two stages are tuned for different things. Stage one is tuned for recall: getting the right chunk into the shortlist at all. Stage two is tuned for precision: getting it to the top of the shortlist. Measure them separately, because they fail separately. Recall@50 (how often the correct chunk appears anywhere in the top 50) tells you whether stage one is even giving the reranker a chance. No reranking metric means anything until that number is high.
\nThe lowest-friction way to add reranking is the CrossEncoder class from sentence-transformers. The ms-marco-MiniLM models are small, run fine on CPU for modest shortlists, and are trained for exactly this scoring task.
The rerank function below scores each (query, candidate) pair, attaches that score to the candidate, and returns the top_n highest.
from sentence_transformers import CrossEncoder\n\n# small, fast, trained on MS MARCO passage ranking\nreranker = CrossEncoder(\"cross-encoder/ms-marco-MiniLM-L6-v2\")\n\ndef rerank(query, candidates, top_n=5):\n # candidates: list of dicts from stage one, each with a \"text\" field\n pairs = [(query, c[\"text\"]) for c in candidates]\n scores = reranker.predict(pairs) # one score per pair\n for c, s in zip(candidates, scores):\n c[\"rerank_score\"] = float(s)\n ranked = sorted(candidates, key=lambda c: c[\"rerank_score\"], reverse=True)\n return ranked[:top_n]Wire it in between retrieval and prompt assembly. Stage one returns 50 candidates, and stage two cuts them to 5:
\ncandidates = hybrid_search(query, k=50) # stage one: wide net\ntop = rerank(query, candidates, top_n=5) # stage two: precise\ncontext = \"\\n\\n\".join(c[\"text\"] for c in top)\nanswer = llm.generate(build_prompt(query, context))predict batches internally, so the 50 pairs go through as a handful of batches, not 50 separate calls. On a MiniLM model, a shortlist of 50 short chunks takes tens of milliseconds on a GPU and a few hundred on CPU. Measure that number on your own hardware before you decide how big a shortlist you can afford.
If you do not want to run the model yourself, the reranker is one of the few RAG components where a hosted API is genuinely convenient, because it is stateless. You send the query and the candidate texts, and you get back scores. Cohere’s Rerank endpoint is the common choice. The function below has the same inputs and outputs as the local version:
\nimport cohere\n\nco = cohere.ClientV2()\n\ndef rerank_hosted(query, candidates, top_n=5):\n docs = [c[\"text\"] for c in candidates]\n resp = co.rerank(\n model=\"rerank-v3.5\",\n query=query,\n documents=docs,\n top_n=top_n,\n )\n # results come back sorted, each with an index into `docs`\n return [\n {**candidates[r.index], \"rerank_score\": r.relevance_score}\n for r in resp.results\n ]The API returns results already sorted, each with a relevance_score and an index back into the list you sent, which the code uses to recover the original candidate. rerank-v3.5 has a 4096-token context per document and truncates longer ones, a detail worth remembering when your chunks are large.
The tradeoff against the local model is the usual one. You have no GPU to manage and no model to host, but you pay per call and add a network hop to your critical path.
\nOne knob decides both quality and latency: k, the number of candidates you hand the reranker.
k of 5 would have missed it.Don’t guess. Retrieve a generous number, say 100, and measure against a set of real queries how often the correct chunk falls inside the top k for a range of k values. You are looking for the point where recall stops climbing. If recall@50 and recall@100 are the same, there is no reason to rerank 100; the extra 50 are pure cost.
In practice, a k somewhere between 20 and 50 covers most of the recall for a well-tuned first stage, and reranking down to the top 3 to 5 is plenty for the prompt.
Reranking a bad shortlist. This is the one that wastes the most time. If retrieval never surfaced the right chunk, the reranker reorders garbage, and you get a confidently wrong answer with a high rerank score. When a query fails, check whether the answer chunk is anywhere in the pre-rerank candidates before you touch the reranker. If it is not, the bug is upstream, in retrieval or in chunking, and no reranker will save it.
\nTruncation on long chunks. Like any transformer, a cross-encoder has an input limit, and the query plus a long document can blow past it. When that happens, the model silently reads only a prefix of the chunk. If the relevant sentence lives in the tail, the reranker scores the pair as if that sentence were not there. Keep chunks inside the reranker’s context window, or the model is grading a document it only half read.
\nTreating the score as a probability. The output is a relevance score, not a calibrated confidence, and the scale differs between models. A threshold you tuned for MiniLM-L6 means nothing for rerank-v3.5. If you want a cutoff that drops weak candidates, calibrate it per model against your own data, and re-check it whenever you swap the model.
Latency you forgot to budget. Reranking adds a synchronous step between retrieval and generation. It is small on its own, but it lands right before the model call, so it stacks on top of the slowest part of the request. If you stream responses (I covered SSE (Server-Sent Events) streaming from FastAPI in an earlier post), the rerank has to finish before the first token can go out. The user waits through it with nothing on screen. Measure it as part of time-to-first-token, not as a free add-on.
\nThe mistake I would warn against is adding a reranker before you have measured whether retrieval is your actual bottleneck. It is a satisfying component to bolt on, and it does help. But if your recall@50 is already poor, the reranker cannot do anything with candidates that do not contain the answer.
\nMeasure the first stage first. Fix chunking and retrieval until the right document reliably lands in the top 50, and only then add reranking to sharpen the order. Done in that sequence, the reranker is the last few points of precision. Done first, it is a distraction that hides the real problem behind a plausible-looking score.
\nReranking is the stage that decides which retrieved chunks actually reach the model. It is the difference between a copilot that finds the right transfer-error fix and one that returns four paragraphs about transfer errors in general.
\nThis is the third piece I have written about the retrieval pipeline behind Archi, the RAG copilot I worked on for CMS computing operations at CERN. It follows chunking, which sets the ceiling, and hybrid search, which widens the net. The order matters: chunk so the answer stays whole, retrieve so it makes the shortlist, rerank so it reaches the prompt. Get all three right and the model finally sees what it needed.
\nDiagrams by M. Hassan Ahmed, released under CC0. Image credit: original work by the author.
","frontmatter":{"title":"Cross-Encoder Reranking for RAG Pipelines","date":"2026-07-08T00:00:00.000Z","description":"Retrieval puts the right chunk at rank 8, but the generator only reads the top few. How a cross-encoder reranker reorders RAG candidates, and where it fails.","thumbnail":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAABzElEQVQoz4WQ6Y6bMBSFUTWZCTvYgLGNzb4kEAgQMglJJn3/p+ptRqpU9UelT9bR9TnHi7IT6bLrvsblMZ7WYboN07U7Pqfz2k9rP8LwfpyB60sDz/lzqvZ7mbVxrmwsYQYly0dZLbI+18dntl+r4Rk3l7y718efxeFe9g9Y092F51NUzBYpIfVhS0V3IsMVuisMV1peAkJ3XrgSJiA0V25NrsMZKNFs2JKmG+v2b79iegXNTsnuxvOFZYvHDwZKoPG7QnOEZUcsK+g65etS3M/xdSZjZ/IU8orLOtncw3R2aYtYh6MB6lSLaXak2pGFBPejKG/E+Sa6mdS9V3aobG1ZqRZXHLoXzYMkJxwdPTECJs5eYW7YXBBpOEzDKc+vtLhgPkA7eODZ4FHcoIybG00mzFqPd4CJJGyE2D+lZE5DirEB4fLKy0/EO8RaFB0sL9NMqhh+FRZ3LCabHhzWA/BDGyMMMJOEOy41HQpdBoqdILf93A4K0JrNVIsqBY+/+nYoq1rKWohGSh9z5NLQ5+9G+GGEWzNUv7Go+ke/UBgmO8ELmcSUyZAyzw8Q8RD4yN+E/2rlTQ9+aORNIxs9hNu+6eRdDzZ6sDX+zy/csEbw2u67jwAAAABJRU5ErkJggg==","aspectRatio":1.899441340782123,"src":"/static/8653045cf4c590d50d2952d56a05590d/40a76/hero.png","srcSet":"/static/8653045cf4c590d50d2952d56a05590d/c972b/hero.png 340w,\n/static/8653045cf4c590d50d2952d56a05590d/27625/hero.png 680w,\n/static/8653045cf4c590d50d2952d56a05590d/40a76/hero.png 1360w,\n/static/8653045cf4c590d50d2952d56a05590d/ed396/hero.png 2000w","sizes":"(max-width: 1360px) 100vw, 1360px"}}}}}},"pageContext":{"slug":"/2026-07-08-cross-encoder-reranking-rag/","previous":"blog/2026-07-07-kubernetes-liveness-readiness-startup-probes/","next":"blog/2026-07-10-fastapi-blocking-event-loop/"}},"staticQueryHashes":["32046230"]}