{"componentChunkName":"component---src-templates-tag-page-js","path":"/tags/data-engineering/","result":{"data":{"site":{"siteMetadata":{"title":"M.Hassan Ahmed","author":"Hassan11196"}},"allMarkdownRemark":{"totalCount":1,"edges":[{"node":{"excerpt":"The first time retrieval quality on Archi dropped for no obvious reason, the cause turned out to be embarrassingly mundane: duplicates. The…","fields":{"slug":"/2026-09-19-minhash-lsh-near-duplicate-detection-rag/"},"frontmatter":{"date":"2026-09-19T00:00:00.000Z","title":"Dedupe a RAG Corpus with MinHash and LSH","description":"Exact md5 hashing misses near-duplicate documents; comparing every pair is O(n²). How MinHash and LSH find and cluster them across a RAG corpus.","tags":["RAG","LLM","Python","Search","Data Engineering"],"thumbnail":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAACDUlEQVQoz02PW2+bMBiGuVhOgDEHGxswhnAMJKQkKLRJIUnTbqu0dZo0bVOlSpN2sV3tF+zPz+m0atJj6/2Ory2VGStSbz7zT/um3y537bzbVoeuPh2aw3UtwtentttV15eLY7/qrqoy94uMlc9IIy0cggCSYlbfpFUfz7t0sRe62rwp6lNUdvP1XTzvk0VfrE5ptddJOQBcTAkk1eQCYAWyzgR/Q8XwhVYMIbisc8UMZSMQWjUDgMJz/3ObJGoTTYz5qhGImuXkNpubNJMhkzVnrDnISVgQO35ikhS7M6AHE+CMIFN0X/KSTV7f2MHKdCud5JjXdriGJEPBJjv+rJuv3z9sH59+3z7+KHef+fEBtJfl7be3ix6argTtxKA5wImGE/E8RLNiegGsqYaiMS5cP46ikLI8jGPKogmOFDsEJDFQJENPcqMmXe6j8tpiS4BTxJZuuNZwavhNd3Gq63df7tv797/6j49V+7CqP4XlvmmfdkWr6Y4ESWo6M2EunCGOkJOLD5t2Au2M+wvCLqoi6bZXSdUgf0XZ2nBKyi8ZzRToSGOVvKDpLsKBafmWxRWNvpLxREU68jgnGNtAs6BhQh3phgF0PAFUEueFMSBDxR6pZ8aAyiKjkjz113VaFZ7j8ek05EGYxKHrehNAhLP9H0TM/7vPGbFFBhhA4WMPFTpQ6FAhA5kO5HPDH8eyT3LwxOXyAAAAAElFTkSuQmCC","aspectRatio":1.899441340782123,"src":"/static/4aa6b11a00da437300ba4f3109b953d1/40a76/hero.png","srcSet":"/static/4aa6b11a00da437300ba4f3109b953d1/c972b/hero.png 340w,\n/static/4aa6b11a00da437300ba4f3109b953d1/27625/hero.png 680w,\n/static/4aa6b11a00da437300ba4f3109b953d1/40a76/hero.png 1360w,\n/static/4aa6b11a00da437300ba4f3109b953d1/ed396/hero.png 2000w","sizes":"(max-width: 1360px) 100vw, 1360px"}}}}}}]}},"pageContext":{"tag":"Data Engineering"}},"staticQueryHashes":["32046230"]}