Consistent Hashing: Sharding Without the Reshuffle
Add a cache node with plain modulo hashing and almost every key moves. Consistent hashing moves only one arc. How the hash ring and virtual nodes work.
#Caching(2)
How to Build a Semantic Cache for LLM Apps
Exact-match caching misses paraphrases, so LLM bills stay high. Here is how to build a semantic cache with embeddings, a similarity threshold, and its traps.