Lexical & Hybrid Retrieval

Most teams start with embeddings. The retrievers that do best in production usually run two systems, one matching words and one matching meaning, and combine the results. This chapter explains why neither is enough alone and how to merge them without tuning any weights.

Two retrievers, two blind spots

Lexical retrieval (BM25 is the standard) scores documents by the exact words they share with the query, giving rare words more weight than common ones. Dense retrieval embeds the query and documents and ranks by cosine similarity, matching on meaning. They fail on different kinds of query:

Lexical search handles exact terms: identifiers, error codes, rare jargon, part numbers and names. Dense search handles paraphrase, where the user's words differ from the document's but the meaning matches. Most real query logs contain both.

What hybrid buys you

Here's a typical result on a 50-query labeled set over a mixed technical corpus, measured as recall@10:

The hybrid score is higher than either input, not somewhere between them. That's because the two retrievers miss different queries. Fusion keeps what each one found, so together they cover more than the better one alone.

Reciprocal rank fusion

You could combine the two with a weighted sum of scores, but BM25 scores and cosine scores sit on different scales, so the weight has to be retuned for every corpus. Reciprocal rank fusion (RRF) ignores the scores and uses only each document's rank in each list:

RRF(d) = Σ  1 / (k + rank_i(d))     for each retriever i that ranked d
         i                              k = 60 by convention

Being near the top of any list earns a document a good score: rank 1 adds 1/61, while rank 50 adds only 1/110. Because the contributions are added up, a document near the top of both lists gets the biggest boost. Ranks mean the same thing in both lists even though the raw scores don't, which is why there's nothing to normalize or tune.

Here's RRF settling a disagreement on "XR-500 firmware crash on launch," a query that's half exact ID and half paraphrase:

All three end up in the top three. BM25 alone would have put the wrong product, C, above B. Dense alone would have put B first and buried A. With fusion, each retriever covers the other's weak spot.

Starter code

from rank_bm25 import BM25Okapi
import re

def toks(s): return re.findall(r"[a-z0-9]+", s.lower())

CORPUS = [
    "XR-500 firmware v4 release notes and changelog",          # 0  exact-ID
    "Resolving crash-on-startup and boot failures",            # 1  paraphrase
    "XR-400 firmware overview",                                 # 2  distractor
    "How to streamline employee onboarding",                   # 3  paraphrase
    "Onboarding checklist: keyword-matchable steps for hires", # 4  lexical
    "SELECT ... GROUP BY returns aggregate error",             # 5  exact-code
    "Improving the new-hire experience and reducing friction", # 6  paraphrase
    "Vault /export endpoint rate limit reference",             # 7  exact-term
]

# (query, index of the ONE correct doc)
GOLD = [
    ("XR-500 firmware",                       0),   # exact ID  -> lexical should win
    ("app keeps crashing when it starts",     1),   # paraphrase -> dense should win
    ("make onboarding easier for new hires",  6),   # paraphrase -> dense
    ("GROUP BY error",                        5),   # exact code -> lexical
    ("export endpoint rate limit",            7),   # exact term -> lexical
]

bm25 = BM25Okapi([toks(d) for d in CORPUS])

def lexical_ranks(query):
    scores = bm25.get_scores(toks(query))
    return sorted(range(len(CORPUS)), key=lambda i: scores[i], reverse=True)

# Toy "dense" retriever: cosine over hashed word-set vectors. Not a real embedding
# model — but it rewards semantic *overlap of concepts* enough to stand in for one
# offline, and keeps the lab free + reproducible. (Production: a real embedding model.)
def dense_ranks(query):
    def vec(s): return set(toks(s))
    q = vec(query)
    def sim(i):
        d = vec(CORPUS[i])
        return len(q & d) / ((len(q) * len(d)) ** 0.5 or 1)
    return sorted(range(len(CORPUS)), key=sim, reverse=True)

# TODO Step 2: def rrf(rank_lists, k=60) -> fused ranking (list of doc indices)
# TODO Step 3: def recall_at(ranker, k=5) -> fraction of GOLD whose doc is in top-k

def rrf(rank_lists, k=60):
    fused = {}
    for ranks in rank_lists:                     # ranks = [doc_idx in rank order]
        for position, doc in enumerate(ranks):   # position is 0-based
            fused[doc] = fused.get(doc, 0) + 1.0 / (k + position + 1)
    return sorted(fused, key=fused.get, reverse=True)

def recall_at(ranker, k=5):
    hits = sum(gold_doc in ranker(q)[:k] for q, gold_doc in GOLD)
    return hits / len(GOLD)

lex   = lambda q: lexical_ranks(q)
den   = lambda q: dense_ranks(q)
hyb   = lambda q: rrf([lexical_ranks(q), dense_ranks(q)])

print(f"lexical recall@5: {recall_at(lex):.2f}")
print(f"dense   recall@5: {recall_at(den):.2f}")
print(f"hybrid  recall@5: {recall_at(hyb):.2f}")