Maximal Marginal Relevance for RAG

Top-k vector search keeps returning near-duplicate chunks that fill the context window. How Maximal Marginal Relevance reranks for relevant, diverse results.

An operator asks the copilot “what is the retry limit before a transfer job gives up?” Retrieval does its job. The top five chunks come back, all scoring above 0.88, and the answer is right there. The problem is that it is there five times:

  • one chunk says “the retry limit is 3,”
  • the next says “retries cap at 3,”
  • a third is a JIRA comment where someone pasted the same line,
  • and the last two are older revisions of the same runbook.

Five slots in the context window, one fact. Meanwhile, the chunk that said “after the third failure it waits 30 seconds before alerting” never made the cut. It scored 0.82 and got pushed out by the fourth paraphrase of something the model already knew. The retrieval worked. The selection was the failure.

This post is for engineers running a RAG (retrieval-augmented generation) pipeline who keep seeing redundant results eat their top-k, and who want the model to get coverage instead of the same sentence repeated. The fix is a reranking step called Maximal Marginal Relevance (MMR), from a 1998 SIGIR paper by Carbonell and Goldstein. It is old and about 15 lines of code. I wish I had reached for it sooner while building the retrieval side of Archi, the RAG copilot for CMS operations at CERN, where the corpus is full of duplicated text by nature.

I cover why top-k favors duplicates, how MMR scores candidates, a NumPy implementation, how to choose its one parameter, and where it bites.

Why top-k rewards redundancy

Vector search ranks each candidate independently. It computes the cosine similarity between the query embedding and every document embedding, sorts, and hands you the top k. Nothing in that process checks whether the results resemble each other. It only checks whether each one resembles the query.

That is fine when your corpus is clean. It falls apart the moment you have near-duplicates, and most real corpora are full of them:

  • Documentation gets copied between pages.
  • The same error message shows up in forty tickets.
  • A runbook exists in three revisions because nobody deleted the old ones.

Every one of those near-copies scores about the same against a given query, so they arrive as a block and crowd out everything else. The ranking is working exactly as designed. The design just has no notion of “I already have this.”

The cost is real because the context window is finite. If three of your five retrieved chunks say the same thing, you paid three chunks’ worth of tokens for one chunk’s worth of information. You also starved the model of the two facts that would have made the answer complete.

The idea: relevance minus redundancy

MMR reranks a candidate pool by scoring each document against two things at once: how relevant it is to the query, and how similar it is to the documents you have already selected. It picks greedily:

  1. Take the single most relevant chunk first.
  2. For each remaining candidate, subtract a penalty for how much it overlaps with what is already in your set, and take the new winner.
  3. Repeat until you have k.

The formula is one line:

MMR = argmax [ λ · sim(d, query) − (1 − λ) · max sim(d, d_selected) ]
       d ∉ S

sim(d, query) is the relevance term you already had. The second term is the new part. For candidate d, find its similarity to the nearest document already in the selected set S, and subtract it. A chunk that looks just like something you already picked carries a big penalty and loses, even if its raw relevance is high. λ (lambda) is the dial between the two terms, from 0 to 1.

Selecting three chunks from one candidate pool. A1, A2, A3 are near-duplicate paraphrases of the retry limit, each scoring around 0.9; B covers backoff timing at 0.82 and C covers alerting at 0.79. Top-k by relevance returns A1, A2, A3, the same fact three times, and B and C never make the cut. MMR with lambda 0.6 returns A1, B, C: three different facts. A caption notes MMR keeps the most relevant hit, then penalizes anything that looks like what it already picked.

Walk through what happens after the first pick. A1 (“retry limit is 3”) is the most relevant, so it goes in. Round two then scores every remaining candidate against the query and against A1. A2 and A3 are paraphrases of A1, so their redundancy penalty is huge and their net score collapses, even though they out-score B and C on raw relevance. B is about backoff timing, which barely overlaps with A1. It keeps almost all of its relevance score and wins the round.

Round two of MMR selection with A1 already chosen and lambda 0.6. A horizontal bar chart shows, for each candidate, the net MMR score in blue and the redundancy penalty in pink. A2 and A3 have large penalties for being duplicates of A1, dropping their net score to about 0.16. B keeps a small penalty and scores 0.40, winning the round; C scores 0.36. The takeaway: A2 and A3 score higher on raw relevance but the penalty for looking like A1 sinks them, and B wins despite being less relevant because it adds something new.

Implementing MMR in NumPy

You do not need a library for this. Over-fetch a candidate pool from your vector store, then rerank it in memory. If all your vectors are L2-normalized (scaled to length 1), cosine similarity is just a dot product, which keeps this to a couple of matrix operations. The function below computes both similarity terms up front, then runs the greedy loop:

import numpy as np

def mmr(query_vec, doc_vecs, lambda_=0.6, k=5):
    """Rerank doc_vecs by Maximal Marginal Relevance.

    query_vec: (d,)    a single, L2-normalized query embedding
    doc_vecs:  (n, d)  L2-normalized candidate embeddings
    returns:   indices into doc_vecs, in selection order
    """
    sim_to_query = doc_vecs @ query_vec      # (n,)   relevance term
    sim_between = doc_vecs @ doc_vecs.T       # (n, n) pairwise overlap

    # the most relevant chunk is always the first pick
    first = int(np.argmax(sim_to_query))
    selected = [first]
    candidates = [i for i in range(len(doc_vecs)) if i != first]

    while candidates and len(selected) < k:
        # for each candidate, how close is it to its nearest selected doc?
        redundancy = sim_between[np.ix_(candidates, selected)].max(axis=1)
        scores = lambda_ * sim_to_query[candidates] - (1 - lambda_) * redundancy
        winner = candidates[int(np.argmax(scores))]
        selected.append(winner)
        candidates.remove(winner)

    return selected

The important line is sim_between[np.ix_(candidates, selected)].max(axis=1). For every remaining candidate, it takes the maximum similarity to anything already chosen, not the average. That matters because a chunk is redundant if it matches even one thing you already have. Averaging would let a chunk that duplicates A1 slip through by being unlike B and C. The max is the whole point.

In a pipeline, MMR sits between retrieval and the prompt. Pull a generous candidate pool, say 50 chunks, then let MMR select the 5 that go to the model:

candidates = vector_store.search(query_vec, top_k=50)   # over-fetch
vecs = np.array([c.embedding for c in candidates])
chosen = mmr(query_vec, vecs, lambda_=0.6, k=5)
context = [candidates[i].text for i in chosen]

Choosing lambda

The two extremes show what the dial does:

  • λ = 1 turns MMR off. The redundancy term drops out, and you are back to plain top-k.
  • λ = 0 ignores relevance entirely and selects for pure diversity. It will happily hand the model chunks that have nothing to do with the question.

Neither extreme is useful. In practice the interesting range is narrow, roughly 0.5 to 0.8, and the right value depends on how duplicate-heavy your corpus is.

I would not guess at it. Pick a starting point that leans toward relevance, λ = 0.7, and then measure. This is exactly the kind of change that feels like an improvement and occasionally isn’t, which is why I built a retrieval eval before touching it. If you do not have one yet, measuring RAG retrieval quality is the prerequisite for tuning anything here honestly. Turn the dial and run the eval set. Keep the value that actually lifts answer quality, not the one that produces prettier-looking result lists.

Where it bites

MMR can only diversify what you retrieved. If your candidate pool is the same size as k, there is nothing to rerank and MMR is a no-op. The diversity has to exist in the pool first, so over-fetch by a healthy multiple: pull 10x what you intend to keep. If your vector search returned only near-duplicates, MMR will dutifully hand you the least-bad subset of them, which is not the same as finding the missing fact. That missing fact is usually a retrieval or chunking problem, not a selection one.

It reduces redundancy by degree; it does not dedupe. MMR is not exact-match deduplication. Two genuinely identical chunks both still appear in the candidate pool, and MMR only guarantees the second one is penalized, not excluded. If you have literal duplicates in your index, the right fix is to not index them twice. I have argued before that a lot of retrieval pain is really ingestion pain wearing a disguise. Duplicate documents are a clean example: dedupe at ingest, and MMR has less work to do.

Diversity can bury a single-fact answer. For a question whose answer lives in exactly one chunk, pushing λ too low can demote that chunk in favor of “diverse” but less relevant material. MMR helps coverage questions (“what are the ways a job can fail”) far more than pinpoint lookups (“what is the exact retry count”). For aggregation queries where you genuinely want every variant, like “list all the error codes we have seen,” diversity is the wrong objective, and you should skip MMR entirely.

Decide where it sits relative to reranking. A cross-encoder reranker (a model that reads the query and a chunk together to score relevance) gives better relevance scores than raw embedding cosine, but it says nothing about redundancy. The clean composition uses the reranker’s score as the relevance term in the MMR formula and keeps embedding cosine for the redundancy term. Avoid any order where the reranker re-sorts purely by relevance after MMR has diversified, which quietly undoes the diversity you just bought.

Compute cost is not one of the things that bites. MMR is O(N²) in the size of the candidate pool, because of the pairwise similarities. But N is in the tens, so it costs microseconds next to the vector search and the model call. You can reason about MMR purely on quality; the performance side is free.

Closing

MMR is worth knowing because it targets a failure that looks like a retrieval problem and isn’t. When the model’s answer is thin despite good-looking search results, the reflex is to blame the embedding model or the chunker. Sometimes the truth is simpler: retrieval found the right material, then spent the context window on three copies of it.

That pattern is everywhere in the corpus behind Archi. Operations knowledge at CERN lives in wikis, JIRA tickets, and shift logbooks, and the same error message gets pasted verbatim across dozens of tickets. Left to plain top-k, a query about that error returns the same paragraph over and over. MMR is the cheap reranking layer that makes the retrieved set earn its tokens. It pairs naturally with the hybrid search and reranking steps around it. Like most of the useful parts of a RAG pipeline, it is less a clever algorithm than a small correction to a default that was quietly working against you.

Diagrams by M. Hassan Ahmed, released under CC0. No external image was used for this post; the figures are original work by the author.