Embeddings & Dense Retrieval

An embedding turns text into a point in space where nearby points mean similar things. That idea powers semantic search, and it fails in specific, predictable ways you should be able to name before relying on it.

The word-overlap retriever from Chapter 1 can't connect "app keeps crashing" with a page titled "resolving startup failures," because they share no words. Embeddings can. An embedding model maps a piece of text to a list of a few hundred to a few thousand numbers, trained so that texts with similar meanings end up close together. Retrieval becomes geometry: embed the query, find the nearest chunk vectors and return them.

Cosine similarity

Closeness is usually measured with cosine similarity, the cosine of the angle between two vectors. It runs from 1 (same direction, same meaning) through 0 (unrelated) to −1 (opposite). It ignores the length of the vectors, which is what you want, because a passage's meaning shouldn't depend on how long it is.

cos(a, b) = dot(a, b) / (norm(a) * norm(b))     # 1 = same meaning, 0 = unrelated

Approximate nearest neighbors

Comparing a query with ten million chunks one by one is too slow. Production systems use approximate nearest neighbor (ANN) indexes, most often HNSW graphs, which find close vectors by walking a graph instead of checking everything. You give up a little recall for a speedup of 100 to 1,000 times. Vector databases such as pgvector, Pinecone, Qdrant and Weaviate all work this way. You'll rarely build ANN yourself, but you will tune it: searching more of the graph (a higher ef_search) improves recall and adds latency.

Where dense retrieval fails

Starter code

import numpy as np, re

USE_REAL_API = False   # True -> swap embed() for a real embeddings API; harness unchanged

CORPUS = [
    "Resolving crash-on-startup and boot failures",     # 0
    "XR-500 firmware v4 release notes",                  # 1
    "XR-400 firmware v2 release notes",                  # 2
    "Streamline employee onboarding for new hires",      # 3
]

# Toy deterministic 'embedding': hashed bag-of-words into a fixed vector.
# Rewards concept overlap (stands in for semantics), and — like a real model —
# maps XR-500 and XR-400 to *near* vectors because their forms overlap.
DIM = 64
def embed(text):
    v = np.zeros(DIM)
    for w in re.findall(r"[a-z0-9]+", text.lower()):
        v[hash(w) % DIM] += 1.0
        # blur digits so "xr" dominates and 500/400 look similar (mimics ID blurring)
        if w.isalnum() and any(c.isdigit() for c in w):
            v[hash(re.sub(r'\d','#',w)) % DIM] += 1.5
    return v

def cosine(a, b):
    return float(a @ b) / ((np.linalg.norm(a)*np.linalg.norm(b)) or 1)

DOCVECS = [embed(d) for d in CORPUS]

def top1(query):
    qv = embed(query)
    sims = [cosine(qv, dv) for dv in DOCVECS]
    i = int(np.argmax(sims))
    return i, CORPUS[i], round(sims[i], 3)

# TODO: run top1() for a paraphrase query and an exact-ID query; print both.

print("paraphrase:", top1("my app won't boot"))       # -> doc 0, the startup-failures doc
print("exact ID:  ", top1("XR-500 firmware"))          # -> may return doc 2 (XR-400!), not doc 1