![\"Pipeline](\"/276cea2569264485a770c05cfae8219c/pipeline.svg\")
The first time retrieval quality on Archi dropped for no obvious reason, the cause turned out to be embarrassingly mundane: duplicates. The crawler that indexes CERN’s internal portals, JIRA tickets, and operator logbooks was doing its job a little too well. It pulled in:
\nNone of these were byte-for-byte identical, so an md5 hash of each document came out different. As far as the index knew, they were four separate sources of truth.
\nThat is the problem this post tackles. When a corpus is full of near-duplicates, retrieval spends its top-k slots (the k best-scoring results) returning the same content in slightly different clothes. The reranker wastes effort ordering copies of one answer, and the context window fills with redundancy instead of coverage. This post is for anyone building an ingestion pipeline for RAG (retrieval-augmented generation) or search who has more documents than they can eyeball. You have probably noticed that “just hash it” quietly fails the moment two documents are 98% the same instead of 100%.
\nThe tools that fix this are MinHash and Locality-Sensitive Hashing (LSH). They are old: MinHash comes from Andrei Broder’s work on deduplicating the AltaVista web index in the late 1990s (Broder, 1997). They hold up because the problem never went away. Below I go through the pipeline in order: turning documents into sets, compressing the sets into MinHash signatures, using LSH to find candidate pairs, and turning those pairs into clean clusters.
\nExact hashing is the obvious first move, and it catches exactly one case: documents that are identical down to the last byte. Change a single character (a trailing newline, a re-rendered timestamp, a Fwd: in the subject) and the hash is completely different. Cryptographic hashes are designed to behave this way. It is the opposite of what you want when the question is “are these two documents basically the same thing?”
So you reach for a similarity measure. The natural one for text is the Jaccard index. Represent each document as a set of features, then measure the overlap as the size of the intersection divided by the size of the union:
\nJ(A, B) = |A ∩ B| / |A ∪ B|If two documents share most of their features, Jaccard is close to 1. If they share almost nothing, it is close to 0. That is exactly the “basically the same” signal that exact hashing throws away.
\nThe catch is cost. Computing Jaccard for every pair of documents takes O(n²) comparisons, and each comparison touches two full feature sets. For a few thousand documents that is fine. A real ops corpus accumulates hundreds of thousands, and at that size n² means tens of billions of set operations; you will not finish before the next crawl starts. MinHash and LSH exist to get the answer of that O(n²) scan without paying for it.
\nBefore any hashing, you need the sets. The standard construction is shingling (also called k-grams): slide a window of k consecutive tokens across the document and collect every window. For example, the set of word 3-grams for “the disk was full again” is {"the disk was", "disk was full", "was full again"}. The function below builds that set from lowercased, whitespace-split words, and returns the whole text as one shingle when it is shorter than k.
def shingles(text: str, k: int = 3) -> set[str]:\n tokens = text.lower().split()\n if len(tokens) < k:\n return {\" \".join(tokens)}\n return {\" \".join(tokens[i : i + k]) for i in range(len(tokens) - k + 1)}The choice of k is the first real tradeoff:
k=1, individual words) make almost every document overlap, because English documents share common words. Unrelated texts get high Jaccard scores.k=8) only match on long verbatim runs, so a paraphrase or a reformatted paragraph looks completely new.For prose I have found word 3-grams to 5-grams a reasonable default. The Mining of Massive Datasets textbook (chapter 3, the definitive treatment of this whole pipeline) suggests picking k large enough that any given shingle has a low probability of appearing in a document. For short, noisy text like log lines, where word boundaries are unreliable, consider character shingles instead.
Whatever you pick, apply it consistently and normalize first. Lowercase the text, collapse whitespace, and strip the boilerplate header and footer that every wiki page carries, all before shingling. Otherwise a shared navigation bar makes two unrelated pages look similar. Boilerplate is the single most common reason a dedup pass produces nonsense, and it deserves a dedicated cleaning step.
\nStoring and intersecting full shingle sets is still expensive. MinHash replaces each set with a short, fixed-length signature, and comparing two signatures estimates their Jaccard similarity directly.
\nHere is the trick. Apply a random hash function to every element of a set and keep the minimum value. For two sets A and B, the probability that they produce the same minimum equals their Jaccard similarity. Repeat with num_perm independent hash functions and each document gets a signature of num_perm integers. The fraction of positions where two signatures agree is then an unbiased estimate of J(A, B), and more permutations give a tighter estimate. (Wikipedia’s MinHash article has the proof if you want to see why the minimum trick works.)
You do not implement this by hand in production. The datasketch library is the well-worn Python choice. The code below builds a MinHash from a text’s 3-gram shingles, then compares two logbook-style sentences that differ only at the end:
from datasketch import MinHash\n\ndef minhash(text: str, num_perm: int = 128) -> MinHash:\n m = MinHash(num_perm=num_perm)\n for sh in shingles(text, k=3):\n m.update(sh.encode(\"utf-8\"))\n return m\n\na = minhash(\"the disk on node 14 was full again this morning\")\nb = minhash(\"the disk on node 14 was full again last night\")\nprint(a.jaccard(b)) # estimated Jaccard, ~0.5 herenum_perm is the dial between accuracy and memory. 128 permutations is a common starting point. The estimate’s standard error falls roughly with the square root of num_perm, so doubling it does not halve the error, and going past 256 rarely earns its cost for dedup. The important win is already in hand: every document is now a fixed signature of 128 integers, however long the original was.
The diagram below shows the whole pipeline, including the LSH stage that comes next.
\n
MinHash shrinks each document, but comparing every signature to every other is still O(n²), just with cheaper comparisons. LSH removes the n² entirely:
\nb bands of r rows each, so that b * r = num_perm.The intuition: two very similar signatures will probably agree across a whole band somewhere. Two dissimilar ones are unlikely to agree on an entire band by chance.
\nThe probability that a pair with Jaccard similarity s becomes a candidate is:
P(candidate) = 1 - (1 - s^r)^bThat function is an S-curve. It stays near zero below a threshold, sits near one above it, and the transition is sharp. The threshold, the point where the probability crosses 0.5, is approximately (1/b)^(1/r). You tune b and r to put that threshold where your definition of “duplicate” lives. The chart below plots the curve for 16 bands of 8 rows.
![\"Line](\"/6d757ce8626afce89c36fafd6143539a/lsh-s-curve.svg\")
In practice you let the library solve for b and r from a target threshold. datasketch does this when you build the index. The code below inserts every document’s signature into an index set for Jaccard of 0.8 or more, then asks for the near-duplicates of one ticket:
from datasketch import MinHashLSH\n\n# Index near-duplicates at Jaccard >= 0.8\nlsh = MinHashLSH(threshold=0.8, num_perm=128)\n\nsignatures = {doc_id: minhash(text) for doc_id, text in corpus.items()}\nfor doc_id, m in signatures.items():\n lsh.insert(doc_id, m)\n\n# For any document, retrieve its near-duplicate candidates in ~constant time\ndupes_of_4021 = lsh.query(signatures[\"ticket-4021\"])query returns candidates in roughly constant time per document, so a full dedup pass over the corpus is close to linear instead of quadratic. That is the difference between a job that finishes during ingestion and one that never runs.
LSH gives you candidate pairs, not final clusters, and you deliberately tune it to over-return slightly. Two steps close the gap.
\nFirst, verify. LSH is only a filter. Confirm each candidate with the actual MinHash estimate (or the exact Jaccard, if you kept the shingle sets) and drop pairs below your real threshold. This removes the small number of false positives that banding lets through.
\nSecond, cluster. In practice duplication is transitive: if A duplicates B and B duplicates C, all three are the same document. Build a graph with the confirmed pairs as edges and take its connected components. A union-find structure does this in near-linear time. Each component becomes one cluster, and you keep a single canonical member. My rule for “canonical” is the longest, most recently updated version, since for an ops runbook the fullest copy is usually the one worth indexing. Everything else is dropped from the index or collapsed into a pointer to the canonical document.
\nSemantic duplicates are invisible to this. MinHash measures lexical overlap, meaning shared wording rather than shared meaning. “The node ran out of memory” and “the machine hit an OOM (out-of-memory) condition” mean the same thing but share almost no shingles, so their Jaccard is near zero and LSH will never pair them. That is not a bug; it is the boundary of the technique. Same-meaning, different-words duplicates are a job for embeddings and vector similarity, a separate pass with different tradeoffs that I covered in choosing an embedding model for RAG. Use MinHash for the cheap lexical dupes and embeddings for the semantic ones; they solve different halves of the same problem.
\nShort documents are unstable. A one-line log message has very few shingles, so changing a single word swings its Jaccard wildly. Below some length the estimate is too noisy to trust. Set a minimum token count, and fall back to exact-match or normalized-string comparison for anything shorter.
\nBoilerplate dominates similarity if you let it. This is the one that will actually burn you. If every page shares a 200-word footer, two otherwise unrelated pages can clear an 0.8 threshold on the footer alone. Strip templated chrome before shingling, and when Jaccard scores come back suspiciously high for obviously different documents, suspect boilerplate first.
\nThe threshold is a product decision, not a default. An 0.8 threshold treats “80% overlapping” as duplicate. Whether that is right depends on whether your corpus holds legitimately similar-but-distinct documents; two release notes for consecutive versions might be 85% identical and both worth keeping. Sample the pairs near your threshold and read them before you trust the number.
\nOn the first pass I treated dedup as a one-time batch job over the whole corpus. That is the wrong shape for a crawler that runs continuously. The MinHashLSH index supports incremental insert, so the better design checks each newly crawled document against the standing index at ingestion time. That is the moment to decide whether it is a new source or a copy of one already indexed. It is the same instinct behind doing RAG indexing incrementally rather than rebuilding from scratch. I would also persist the signatures instead of recomputing them every run: they are small, and shingling the full corpus again and again is wasted work.
The other thing I underweighted early was measurement. It is tempting to pick a threshold, watch the duplicate count drop, and call it done. But the number that matters is whether retrieval got better. The only way to know is to hold out a labelled query set and measure recall and answer quality before and after, the discipline I leaned on in measuring RAG retrieval quality. A dedup pass that quietly drops a document you needed is worse than the redundancy it removed.
\nNear-duplicate detection is unglamorous plumbing, and it is exactly the kind of thing that decides whether a RAG system feels sharp or sludgy. A deduped corpus returns diverse, non-redundant context. A corpus that has not been deduped spends its best retrieval slots saying the same thing three times. MinHash and LSH let you dedupe at a scale where reading the documents yourself stopped being an option long ago, and they do it in close to linear time.
\nI kept coming back to this pattern across the data plumbing behind Archi and the CMS workflow operations tooling at CERN. The cheap, approximate, old algorithm that gets you 95% of the way for 5% of the cost is usually the one worth reaching for first. Save the expensive semantic pass for the cases the cheap one provably cannot catch.
\nDiagrams by M. Hassan Ahmed, released under CC0. No external image was used for this post; the figures are original work by the author.
","frontmatter":{"title":"Dedupe a RAG Corpus with MinHash and LSH","date":"2026-09-19T00:00:00.000Z","description":"Exact md5 hashing misses near-duplicate documents; comparing every pair is O(n²). How MinHash and LSH find and cluster them across a RAG corpus.","thumbnail":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAACDUlEQVQoz02PW2+bMBiGuVhOgDEHGxswhnAMJKQkKLRJIUnTbqu0dZo0bVOlSpN2sV3tF+zPz+m0atJj6/2Ory2VGStSbz7zT/um3y537bzbVoeuPh2aw3UtwtentttV15eLY7/qrqoy94uMlc9IIy0cggCSYlbfpFUfz7t0sRe62rwp6lNUdvP1XTzvk0VfrE5ptddJOQBcTAkk1eQCYAWyzgR/Q8XwhVYMIbisc8UMZSMQWjUDgMJz/3ObJGoTTYz5qhGImuXkNpubNJMhkzVnrDnISVgQO35ikhS7M6AHE+CMIFN0X/KSTV7f2MHKdCud5JjXdriGJEPBJjv+rJuv3z9sH59+3z7+KHef+fEBtJfl7be3ix6argTtxKA5wImGE/E8RLNiegGsqYaiMS5cP46ikLI8jGPKogmOFDsEJDFQJENPcqMmXe6j8tpiS4BTxJZuuNZwavhNd3Gq63df7tv797/6j49V+7CqP4XlvmmfdkWr6Y4ESWo6M2EunCGOkJOLD5t2Au2M+wvCLqoi6bZXSdUgf0XZ2nBKyi8ZzRToSGOVvKDpLsKBafmWxRWNvpLxREU68jgnGNtAs6BhQh3phgF0PAFUEueFMSBDxR6pZ8aAyiKjkjz113VaFZ7j8ek05EGYxKHrehNAhLP9H0TM/7vPGbFFBhhA4WMPFTpQ6FAhA5kO5HPDH8eyT3LwxOXyAAAAAElFTkSuQmCC","aspectRatio":1.899441340782123,"src":"/static/4aa6b11a00da437300ba4f3109b953d1/40a76/hero.png","srcSet":"/static/4aa6b11a00da437300ba4f3109b953d1/c972b/hero.png 340w,\n/static/4aa6b11a00da437300ba4f3109b953d1/27625/hero.png 680w,\n/static/4aa6b11a00da437300ba4f3109b953d1/40a76/hero.png 1360w,\n/static/4aa6b11a00da437300ba4f3109b953d1/ed396/hero.png 2000w","sizes":"(max-width: 1360px) 100vw, 1360px"}}}}}},"pageContext":{"slug":"/2026-09-19-minhash-lsh-near-duplicate-detection-rag/","previous":"blog/2026-09-20-mixture-of-experts-explained/","next":"blog/2026-09-24-roaring-bitmaps-fast-filters/"}},"staticQueryHashes":["32046230"]}