Embeddings & Dense Retrieval

An embedding turns text into a point in space where "near" means "means the same thing." That one idea powers semantic search — and it fails in specific, predictable ways that you must be able to name before you reach for it. This chapter is the physics of dense retrieval.

A word-overlap retriever (Ch. 1) can't match "app keeps crashing" to a doc titled "resolving startup failures" — zero shared words. Embeddings fix exactly that. An embedding model maps a chunk of text to a vector of a few hundred to a few thousand numbers, trained so that texts with similar meaning land close together and unrelated texts land far apart. Retrieval becomes geometry: embed the query, find the nearest chunk vectors, return them.

Cosine similarity: the one operation

"Near" is measured by cosine similarity — the cosine of the angle between two vectors, ranging from 1 (identical direction, same meaning) through 0 (orthogonal, unrelated) to −1 (opposite). It ignores vector length and cares only about direction, which is what you want: the meaning of "20 requests per minute" shouldn't depend on how long the passage is. The whole retrieval step is: compute cosine between the query vector and every chunk vector, sort, take the top-k.

Cosine, in one line

cos(a, b) = dot(a, b) / (norm(a) * norm(b))     # 1 = same meaning, 0 = unrelated

You don't scan every vector: ANN

Cosine against all 10 million chunks per query is too slow. Production uses Approximate Nearest Neighbor (ANN) indexes — most commonly HNSW (Hierarchical Navigable Small World graphs) — which find the near-neighbors by walking a graph instead of scanning everything, trading a sliver of recall for 100–1000× speed. This is the machinery inside every vector database (pgvector, Pinecone, Qdrant, Weaviate). The engineering knob is the recall/latency trade: search more of the graph (higher ef_search) for better recall at higher latency, or less for speed. You rarely implement ANN; you tune it and know what it's approximating.

Where dense retrieval fails — name these cold

Embeddings are not magic; they have a signature blind spot that Chapter 5 exists to cover. Know exactly where they break:

Starter code

import numpy as np, re

USE_REAL_API = False   # True -> swap embed() for a real embeddings API; harness unchanged

CORPUS = [
    "Resolving crash-on-startup and boot failures",     # 0
    "XR-500 firmware v4 release notes",                  # 1
    "XR-400 firmware v2 release notes",                  # 2
    "Streamline employee onboarding for new hires",      # 3
]

# Toy deterministic 'embedding': hashed bag-of-words into a fixed vector.
# Rewards concept overlap (stands in for semantics), and — like a real model —
# maps XR-500 and XR-400 to *near* vectors because their forms overlap.
DIM = 64
def embed(text):
    v = np.zeros(DIM)
    for w in re.findall(r"[a-z0-9]+", text.lower()):
        v[hash(w) % DIM] += 1.0
        # blur digits so "xr" dominates and 500/400 look similar (mimics ID blurring)
        if w.isalnum() and any(c.isdigit() for c in w):
            v[hash(re.sub(r'\d','#',w)) % DIM] += 1.5
    return v

def cosine(a, b):
    return float(a @ b) / ((np.linalg.norm(a)*np.linalg.norm(b)) or 1)

DOCVECS = [embed(d) for d in CORPUS]

def top1(query):
    qv = embed(query)
    sims = [cosine(qv, dv) for dv in DOCVECS]
    i = int(np.argmax(sims))
    return i, CORPUS[i], round(sims[i], 3)

# TODO: run top1() for a paraphrase query and an exact-ID query; print both.

print("paraphrase:", top1("my app won't boot"))       # -> doc 0, the startup-failures doc
print("exact ID:  ", top1("XR-500 firmware"))          # -> may return doc 2 (XR-400!), not doc 1