{"componentChunkName":"component---src-templates-blog-post-js","path":"/blog/2026-09-11-cosine-dot-product-euclidean-vector-search/","result":{"data":{"site":{"siteMetadata":{"title":"M.Hassan Ahmed","author":"Hassan11196"}},"markdownRemark":{"id":"f283d1b6-cb95-5855-a273-a2b67a3c9b2e","excerpt":"Every vector database asks you to pick a distance metric when you create the index. Most tutorials wave past the choice with “use cosine,” and that works until…","html":"

Every vector database asks you to pick a distance metric when you create the index. Most tutorials wave past the choice with “use cosine,” and that works until the day it quietly doesn’t. You swap in a new embedding model, keep the same index settings, and retrieval gets subtly worse. Long documents start crowding the top results. Or two indexes built from the same corpus return different neighbors, and nobody can say why.

\n

The metric matters, but not in the way the question is usually framed. Cosine similarity, dot product (also called inner product), and Euclidean distance are not three competing ideas of “close.” On normalized vectors they produce the same ranking. The real decision is about what your embeddings look like and what each engine costs you at query time.

\n

This post is for anyone wiring up a RAG (retrieval-augmented generation) retriever or a k-NN (k-nearest-neighbor) index who wants to choose deliberately instead of copying a default. I led the retrieval side of Archi, a RAG copilot for CMS operations at CERN, and this is the mental model I used when someone asked which space_type to set. It covers what each metric measures, why normalization makes them agree, how to match the metric to the model, and how pgvector, OpenSearch, and FAISS each expect you to configure it.

\n

What each metric measures

\n

Take a query vector q and a document vector d, each a list of, say, 768 floats.

\n

Dot product is the raw inner product, sum(q[i] * d[i]). It grows with the angle between the vectors and with their lengths. A vector with a large magnitude can score highly just by being big.

\n

Cosine similarity divides the dot product by both magnitudes: dot(q, d) / (norm(q) * norm(d)). That division cancels magnitude, so cosine measures only the angle between the vectors. Its value lies in [-1, 1].

\n

Euclidean distance (L2) is the straight-line gap between the tips of the two vectors, sqrt(sum((q[i] - d[i])^2)). Smaller means closer. That makes it a distance, not a similarity, so you sort ascending instead of descending.

\n

The difference that trips people up is magnitude. Cosine throws it away; the other two keep it. The diagram below shows where that matters.

\n

\"A

\n

Document A and document B point in exactly the same direction. Cosine sees only the angle, so it scores them as equally relevant to the query. Dot product ranks B above A purely because B is longer.

\n

Suppose your embedding model happens to give longer or more “confident” documents a larger norm. Dot product will then systematically float those documents to the top, whether or not they answer the question. That is the classic footgun: using dot product on embeddings you never normalized.

\n

Normalization makes the three metrics agree

\n

Now normalize every vector to unit length before you store it: divide each one by its own magnitude so norm(v) == 1.

\n

Once every vector sits on the unit sphere, cosine and dot product become the same operation. Cosine is defined as the dot product of normalized vectors, so if the vectors are already normalized, dividing by 1 does nothing. Euclidean distance falls in line too, through one identity worth memorizing:

\n
||a - b||^2 = ||a||^2 + ||b||^2 - 2(a . b)\n            = 2 - 2(a . b)     when |a| = |b| = 1
\n

The identity says that for unit vectors, Euclidean distance squared is just 2 - 2 * cosine. As cosine goes up, Euclidean distance goes down, always. So sorting by Euclidean distance gives exactly the same order as sorting by cosine (Wikipedia’s cosine similarity page spells out this relationship). On normalized vectors, “nearest by L2,” “most similar by cosine,” and “largest dot product” are three names for the same sorted list.

\n

\"A

\n

So “which metric?” is really two questions:

\n
    \n
  1. Are your vectors normalized? If yes, the metrics are interchangeable for ranking, and you pick on cost.
    2. \n
  2. If not, how was the model trained? Un-normalized, the metrics genuinely differ, and you need to match the metric to the model’s training objective.
    3. \n
\n

Use the similarity the model was trained with

\n

Each embedding model is trained with a particular similarity function in its loss, and that is the one you should score with. Most modern sentence-embedding models are trained for cosine, and the model card says so.

\n

The sentence-transformers docs are explicit on this point. When a model outputs unit-length vectors, “dot-product and cosine-similarity are equivalent,” and dot product is preferred because it skips a redundant re-normalization (Semantic Textual Similarity).

\n

A handful of retrieval models, including some dense passage retrievers, are trained for raw dot product. They deliberately encode importance into the vector’s norm. For those models, normalizing throws away signal the model meant to keep.

\n

The rule is boring and reliable: read the model card and use the similarity it was trained with.

\n\n

How pgvector, OpenSearch, and FAISS spell the metric

\n

The metric is an abstract choice, but every engine spells it differently, and a couple of the spellings have sharp edges.

\n

pgvector exposes the metric as query operators rather than a config value:

\n\n

The sign flip exists because Postgres index scans only go ascending. Negating turns “largest dot product” into “smallest number” (pgvector README). It is easy to miss, and forgetting it inverts your results. Both queries below sort ascending, and both return the most similar rows first:

\n
-- cosine distance: smaller is more similar\nSELECT id FROM docs ORDER BY embedding <=> query_vec LIMIT 10;\n\n-- inner product: <#> is NEGATIVE dot product, so ascending sort\n-- still returns the largest-dot-product rows first\nSELECT id FROM docs ORDER BY embedding <#> query_vec LIMIT 10;
\n

OpenSearch takes a space_type on the k-NN field mapping. The values are l2, cosinesimil, and innerproduct. OpenSearch ranks on descending score, so it converts each distance into a score where higher is better. Two details matter here:

\n\n

The mapping below follows that advice: a 768-dimension vector field scored with inner product, which assumes you normalize before indexing.

\n
PUT /docs\n{\n  \"settings\": { \"index.knn\": true },\n  \"mappings\": {\n    \"properties\": {\n      \"embedding\": {\n        \"type\": \"knn_vector\",\n        \"dimension\": 768,\n        \"space_type\": \"innerproduct\"\n      }\n    }\n  }\n}
\n

FAISS offers only METRIC_L2 and METRIC_INNER_PRODUCT. It has no cosine metric at all. The FAISS wiki tells you to normalize the vectors with faiss.normalize_L2 and then use inner product, which is exactly the equivalence from earlier. The whole trick is to normalize in place before both add and search:

\n
import faiss\n\nfaiss.normalize_L2(doc_vecs)          # unit length in place\nindex = faiss.IndexFlatIP(768)        # inner product\nindex.add(doc_vecs)\n\nfaiss.normalize_L2(query_vecs)        # normalize the query too\nscores, ids = index.search(query_vecs, k=10)
\n

The line people forget is normalizing the query. If you normalize the corpus but skip the query, every score gets scaled by the query’s arbitrary magnitude. The ranking within a single query survives that. Cross-query score thresholds, and anything that fuses scores across retrievers, stop meaning anything.

\n

Failure modes I’ve actually hit

\n

Dot product on un-normalized vectors. This is the most common one. Retrieval looks fine on a demo corpus, then long documents dominate in production because their embeddings carry a larger norm. Switch to cosine, or normalize and keep dot product, and the length bias disappears.

\n

Half-normalized indexes. You normalize at index time but not at query time, or you normalize new documents while an older segment of the index went in raw. The index returns plausible-looking neighbors that are wrong, and no error tells you. Normalization has to be a property of the whole pipeline: the same function, on ingest and on query, applied to every vector.

\n

Comparing scores across metrics. Cosine sits in [-1, 1], raw dot product is unbounded, and L2 is a distance where zero is perfect. A “relevance > 0.8” cutoff tuned against a cosine index is meaningless on an L2 index. Thresholds are per-metric, and honestly per-model too.

\n

This bites hardest in hybrid search. Naively adding a cosine score to a BM25 keyword score combines two numbers on incompatible scales. That is one reason rank-based fusion behaves better than adding raw scores together.

\n

Assuming the metric changes recall. On normalized vectors it doesn’t: the ranking is identical. What does change recall is the approximate index structure underneath, like the graph in HNSW. The metric and the ANN (approximate nearest neighbor) algorithm are separate knobs, so swapping cosine for L2 to chase a recall regression is looking in the wrong place.

\n

Tradeoffs and my default

\n

If the metrics rank identically once vectors are normalized, the tiebreakers are cost and clarity. Dot product on pre-normalized vectors is the cheapest to compute at query time: there is no per-query square root or division, because you did that work once at index time. That is why the engines nudge you toward it, and why I default there:

\n

Normalize at index time, store unit vectors, query with inner product.

\n

You get cosine’s ranking, dot product’s speed, and one invariant that is easy to test: every stored vector has norm 1.

\n

There are two exceptions. If a model was explicitly trained for raw dot product, the norm carries meaning, so leave the vectors alone. And if you are on pgvector and would rather not think about the <#> sign convention, <=> cosine distance is perfectly fine. On a small index, the performance gap is not worth a subtle bug.

\n

Looking back at early retrieval work, here is what I would do differently: decide normalization once, at the boundary where embeddings enter the system, and assert it. A single assert abs(np.linalg.norm(v) - 1.0) < 1e-6 on ingest would have caught more retrieval weirdness than any amount of metric-swapping. The metric is a small decision. Whether your vectors are actually normalized, everywhere, is what silently decides whether search works.

\n

The metric question sits downstream of picking the embedding model itself, which I covered in choosing an embedding model for RAG. Get both settings right before you index a few million documents and have to redo it.

\n
\n

Diagrams by M. Hassan Ahmed, released under CC0. No external image was used for this post; the figures are original work by the author.

","frontmatter":{"title":"Cosine, Dot Product, or Euclidean for Embeddings?","date":"2026-09-11T00:00:00.000Z","description":"Cosine, dot product, and Euclidean rank vector search results the same on normalized embeddings, differently otherwise. How to pick and configure one.","thumbnail":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAAByElEQVQozz2R3XKbMBCFuQsgrYQACRAIgUAY/+DYvohjJ+1Fmk7f/426Np3OfLNzNNpztAsBnc7whPoT1vHHH/v+5T5/u4/v7vbd37+7+68Hty+8wiPfXtZONAaENf/gTVb4ze42bt6W08/98dPPb6i3hzuyWz7mw93P11xN/y0BSVuS2li0CM06mlrjTpDaEGqStHW/2PEMeR/yRjbbYXu1/gJ5t7oCkhiEZR2XLmY1l305vEI5EtkT5WS/6OnMEhODFnlfmYPUMxXt6gpi3iBEtIkaUfBikMd3cAv0B+j2dHwVy5W6PTFzJLtYuTh59j/MDZprJGL6mddEUBXmyFILiYGk4WmfyJFKB3qi3Y7Pl0R77AlpiTWImV4BXAODmFZmD4/lW4xjueNyoLiXeMQJOcasSrVv5kvWTEHEKiSEkgicR6NOK9zqMRUCmWWyx1wUGCcKjz2QtVnlWW6DCHCAknDNMrvqVHmKHwKqiJaUG3w2JAUeMQsHec5cvBCFFc1FSBUVNWQGRZxoIlsiDdOO10NSD8J4BAVVLRT4k2q0rM8EaEBeiERCoiKhadkhotvkwy7r59zv1XTI3ExLSytL8ibEzqfrL2TMQD9kW0U8AAAAAElFTkSuQmCC","aspectRatio":1.899441340782123,"src":"/static/253f7d1c56e9e7b3fd4601ef40351ea0/40a76/hero.png","srcSet":"/static/253f7d1c56e9e7b3fd4601ef40351ea0/c972b/hero.png 340w,\n/static/253f7d1c56e9e7b3fd4601ef40351ea0/27625/hero.png 680w,\n/static/253f7d1c56e9e7b3fd4601ef40351ea0/40a76/hero.png 1360w,\n/static/253f7d1c56e9e7b3fd4601ef40351ea0/ed396/hero.png 2000w","sizes":"(max-width: 1360px) 100vw, 1360px"}}}}}},"pageContext":{"slug":"/2026-09-11-cosine-dot-product-euclidean-vector-search/","previous":"blog/2026-09-07-parsing-pdfs-for-rag/","next":"blog/2026-09-10-maximal-marginal-relevance-rag/"}},"staticQueryHashes":["32046230"]}