![\"How](\"/c503aa51a02e13fc56f89304fc9eea26/hll-update.svg\")
How many distinct things are in this stream? Distinct users today, distinct error signatures this hour, distinct workflow IDs that touched a broken storage endpoint. The question sounds trivial until you try to answer it fast. The exact answer requires remembering everything you have already seen, so that you never count it twice, and “remember everything” is the part that does not scale.
\nI hit this on the telemetry side of CMS workflow operations at CERN. The logs from WMCore and its surrounding databases are large enough that a plain COUNT(DISTINCT) over a wide time range either times out or quietly eats a node’s heap. HyperLogLog is the algorithm that sidesteps this. It answers “how many unique?” for billions of items using a fixed slab of memory measured in kilobytes, and its summaries can be combined across shards. The catch is that the answer is an estimate.
This post is for engineers who need distinct counts at a scale where exact counting is no longer an option, and who want to know what they are trading away. It covers why exact counting breaks, how HyperLogLog works, why it merges so well, how to use it in Redis and OpenSearch, and where it misleads you.
\nThe obvious way to count uniques is a set. Walk the stream, drop every item into a hash set, and the size of the set is your answer. It is exact and simple. Its memory cost is the problem: the set must hold one entry per unique value, so it grows with the number of distinct items.
\nTen million unique 20-byte workflow IDs already make a couple hundred megabytes of set, before overhead. Do that for fifty different metrics across a retention window, and you are budgeting real RAM for a question nobody needs a byte-perfect answer to.
\nDistribution makes it worse. When your data lives on many shards, each shard can build its own set. But to get the global distinct count you have to merge those sets, which means shipping every ID to one place and de-duplicating there. You have moved the whole dataset across the network just to count it.
\nThat is the wall. What you want instead is a small, fixed-size summary of a set that you can merge cheaply. That is exactly what a HyperLogLog sketch is. (A sketch is a compact data structure that summarizes a large dataset and answers approximate queries about it.)
\nThe whole family of algorithms rests on a bet about randomness. Run each item through a good hash function, so its output looks like uniform random bits. Then look at how many zero bits the hash starts with:
\n1, so no leading zeros.01.001.A hash that begins with ten leading zeros is a one-in-a-thousand event. If you have seen one, you have probably looked at something on the order of a thousand distinct values.
\nThat is the entire intuition, and it goes back to Flajolet and Martin’s 1985 probabilistic counting work. Track the maximum number of leading zeros you have seen across all items, call it ρ, and 2^ρ is a rough guess at the cardinality (the number of distinct items).
“Rough” is the key word. A single unlucky hash with a long run of zeros throws the estimate off by a factor of two, because the whole count rests on one extreme observation. HyperLogLog’s real contribution is not the leading-zeros idea. It is how it beats down that variance while storing only a few kilobytes.
\nInstead of one running maximum, HyperLogLog keeps many. It splits each hash into two parts:
\nm = 2^p registers (small counters).Each register stores the largest leading-zero count (plus one) that any item routed to it has produced. You are now running m independent little experiments in parallel, and each register is one noisy estimator. The diagram walks through a single update.
The way HyperLogLog combines the registers is where it earns its name. You might expect an ordinary average, but an arithmetic mean is still wrecked by the occasional huge register. Instead, HyperLogLog takes the harmonic mean of 2 raised to each register value, scaled by a correction constant. The harmonic mean is dominated by the small values, so it shrugs off large outliers. The published estimator is:
E = α_m · m² / Σ_j 2^(−M[j])Here M[j] is the value in register j, and α_m is a constant near 0.7213 that corrects the systematic bias of the raw formula. You do not need to memorize the formula.
The property that matters falls out of the formula: with m registers, the standard error is about 1.04 / √m. Quadruple the registers and you halve the error. That relationship is the entire tuning knob, and it explains the numbers you see in real systems. Here is how it plays out in Redis:
m = 16384 registers.1.04 / √16384 = 1.04 / 128, which is 0.0081: the 0.81% error its docs quote.16384 × 6 bits is 12288 bytes: the famous 12KB.That memory is fixed whether you feed the sketch a thousand items or a trillion.
\nWritten out, adding an item is unremarkable, which is the point. Hash the item, use the top bits to pick a register, count leading zeros in the rest, and keep the maximum:
\ndef add(self, item):\n h = hash64(item) # a good 64-bit hash\n j = h >> (64 - self.p) # first p bits -> register index\n w = (h << self.p) & MASK_64 # the rest of the bits\n rho = leading_zeros(w) + 1 # position of the leftmost 1\n self.registers[j] = max(self.registers[j], rho)Every add is constant time and touches one register. There is no growth, no rehashing, no compaction.
\nThe register update is a max, and that one detail is what makes HyperLogLog usable in a distributed system. You can merge two sketches built over two different slices of data by taking the larger of the two values at each register position. The result is bit-for-bit the sketch you would have gotten by feeding both slices into one HyperLogLog from the start.
![\"Merging](\"/5178d5518aa208e606764d44d9348296/hll-merge.svg\")
Two properties of max make this safe:
max is associative and commutative, so neither the order you merge shards in nor how you group them can change the answer.ρ in the same register on both, and max(ρ, ρ) is ρ.This is why a query engine can have each shard compute a tiny local sketch, ship only the sketch to a coordinator, and combine them there. You send twelve kilobytes per shard instead of the whole dataset. Redis exposes this directly as PFMERGE. OpenSearch does it internally every time you run a distributed cardinality aggregation.
You rarely implement HyperLogLog yourself, because the systems you already run ship it.
\nIn Redis, HyperLogLog is three commands, available since 2.8.9. The example adds three IDs (one repeated) to a daily key, reads the count, and rolls two days into a weekly key:
\nPFADD workflows:2026-09-26 wf-4487 wf-4488 wf-4487\nPFCOUNT workflows:2026-09-26 # -> 2, wf-4487 counted once\nPFMERGE workflows:week workflows:2026-09-26 workflows:2026-09-25PFADD folds items into the sketch.PFCOUNT reads the estimate.PFMERGE unions sketches into a new key. That is how you roll daily counts up into a weekly one without ever storing the IDs.In OpenSearch, the same machinery sits behind the cardinality aggregation. It uses HyperLogLog++, Google’s refinement of the original with 64-bit hashes and a sparse layout for small counts. This query counts distinct workflow IDs across the log indexes:
GET logs-*/_search\n{\n \"size\": 0,\n \"aggs\": {\n \"distinct_workflows\": {\n \"cardinality\": {\n \"field\": \"workflow_id\",\n \"precision_threshold\": 3000\n }\n }\n }\n}The precision_threshold is the same error-versus-memory knob from earlier, under a different name. Counts at or below the threshold are near-exact. Above it, you get the HyperLogLog estimate, which OpenSearch documents as typically within about 6% of the true value. The default is 3000 and the maximum is 40000. Raising it buys accuracy with memory, in the same 1.04/√m trade.
For a dashboard panel showing “distinct failing workflows over the last 24 hours,” the default is usually fine. Being off by a few percent on a trending number is not the kind of error that misleads anyone.
\nThe estimate is the honest tradeoff, but a few sharper edges catch people.
\nThe error is relative, not absolute. A 1% standard error on a true count of 50 million is a swing of half a million. That is fine for a trend line and wrong for anything you would put in a billing row or a compliance report. Distinct counts that feed money or audits need exact counting, full stop.
\nA sketch cannot list its members. It counts; it does not enumerate. If the next question after “how many distinct workflows failed?” is “which ones?”, a HyperLogLog has nothing to give you, because it never stored them. Use it for the count, and go back to the source when you need the list.
\nIntersections are a trap. Union is exact and cheap, but there is no direct way to intersect two sketches. The usual workaround is inclusion-exclusion, |A ∩ B| = |A| + |B| − |A ∪ B|. It combines the errors of three separate estimates, so it degrades badly when the sets are similar in size and the overlap is small. If you genuinely need set intersections at scale, HyperLogLog is the wrong tool; something like a MinHash sketch fits better.
Small cardinalities were the original weak spot. The raw estimator is biased low when the true count is small relative to the number of registers. That is precisely why HyperLogLog++ added empirical bias correction and a sparse representation for that range. Redis and OpenSearch already include those fixes. If you hand-roll from the 2007 paper, you will see the low-end skew yourself.
\nEverything rests on the hash. HyperLogLog assumes the hash spreads inputs uniformly across the bit space. A weak hash, or one with structure in the high bits you use to pick registers, breaks that assumption, and the estimate drifts. Use a hash designed for even distribution: not a cryptographic one you picked for other reasons, and not something homegrown.
\nHyperLogLog earns its keep on the dashboard question that would otherwise be a full scan: distinct workflows, distinct users, or distinct error fingerprints over a time range wide enough that exact counting is off the table. It fits naturally into an observability setup. I have written about building OpenSearch dashboards to watch workflow health and about keeping the indexes behind them from filling a disk. Cardinality panels are the natural next thing to put on those dashboards. Now you know the sketch doing the counting and, more usefully, when to stop trusting it.
\nThe pattern generalizes past this one algorithm. A fixed-size, mergeable summary that answers a question approximately is often worth far more than an exact answer you cannot afford to compute. Distinct counts are just the cleanest example.
\nDiagrams by M. Hassan Ahmed, released under CC0. No external image was used for this post; the figures are original work by the author.
","frontmatter":{"title":"HyperLogLog: Count Distinct at Scale in 12KB","date":"2026-09-26T00:00:00.000Z","description":"Counting unique items exactly costs memory that grows with your data. HyperLogLog estimates the cardinality of billions in about 12KB. Here's how it works.","thumbnail":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAAB7ElEQVQoz12RyZKbMBCGOWbACCSBEBZm38xuYxvHYI+XmZrjpHLP+79GGjunVP3V1VL3V7+6JeG8BqG0cPpjfn1E4yV/fyTTNTpdvMMpe7/DMThO2flW3D79YURJAf36k5Jk1ZFVgbDvRduyOVfdpWwveXlK8yHODutqLJpzlO7zaoRqEO90GsoLAZSCVtKCBsiMNStRjUghoUJDhQSIxZhniCWzzBhKMgnedA+qGoNjCJRCXAmxlIoG26Vm5U+tEcssr3WzwXQ7w2ksbwMNzGncaC/8rWakmplrLFN0B+BE52sqqhcMOQh41YT7wnRbnuyIqKxm8O8f7vVuTRMtd8hIZA1gM8F2QZbVi4QIJCTM39jpniVbErYaz42fJ/b7y/r1Jf58s+mMcCTrYoZ1u6CifjnDE0Q6pP3N7yYr7TW3xH6t2Wte7OPr52o38e6IoxYZsYyWkmrGeFnSZU3tigdbHvdG2FG/NsOO57OzlfV4VbFVG9XTMtjq88A5WL4hPjuDIS8Oznhjh5PRH58a+HH07g/3evMeH0a2tb2NE+/ndViZBtGM31TrNXPJvE4Xpb4sDbclqwbegkVNIRE1cWoi6n+LhNFgL7yYnVUmKdhVaaASXyXBf9JZQniGwQq+mvpzGw1fcYHdHwv2F7XYUxz50t48AAAAAElFTkSuQmCC","aspectRatio":1.899441340782123,"src":"/static/d8c408d0f784bb15fc0ce20a2f9339e1/40a76/hero.png","srcSet":"/static/d8c408d0f784bb15fc0ce20a2f9339e1/c972b/hero.png 340w,\n/static/d8c408d0f784bb15fc0ce20a2f9339e1/27625/hero.png 680w,\n/static/d8c408d0f784bb15fc0ce20a2f9339e1/40a76/hero.png 1360w,\n/static/d8c408d0f784bb15fc0ce20a2f9339e1/ed396/hero.png 2000w","sizes":"(max-width: 1360px) 100vw, 1360px"}}}}}},"pageContext":{"slug":"/2026-09-26-hyperloglog-count-distinct-at-scale/","previous":"blog/2026-09-27-rope-position-embeddings-context-length/","next":"blog/2026-09-25-gpu-memory-serve-llm/"}},"staticQueryHashes":["32046230"]}