Evaluating Retrieval

Everything so far — chunking, hybrid, reranking, query transforms — is a knob. Without measurement, turning knobs is superstition. This chapter builds the evaluation harness that turns "I think hybrid helped" into "recall@10 went 0.71 → 0.83 on our 80-query gold set." It's the skill that defines the role.

Here's the uncomfortable truth that separates a RAG engineer from a hobbyist: you cannot improve retrieval you don't measure, and you can't measure it without a gold set. A gold set is a list of realistic queries, each labeled with the chunk(s) that actually answer it. Building one is the least glamorous and most valuable thing you'll do — 50–100 labeled queries is enough to make every downstream decision empirical instead of vibes.

The four metrics you must be able to explain

The one to internalize is recall@k as the retrieval north star: it directly measures the thing that gates everything downstream — whether the answer even reaches the model. A synthesis prompt can't fix a recall miss. So you tune retrieval to maximize recall@k at a k you can afford to rerank (say recall@50), then let reranking convert that into precision@3. The two metrics correspond to the two stages.

Building the gold set without dying

You don't hand-label 100 queries from scratch. The efficient path: take real user queries from logs, retrieve candidates, and have humans (or a strong LLM as a first-pass labeler, human-verified) mark which chunks are relevant. Even 50 queries stratified across your query types — identifier lookups, how-tos, multi-hop — is enough to catch regressions and compare retrievers. Stratification matters more than raw count: 50 queries covering your real distribution beat 500 all of one kind.

Starter code

import math

# gold: query -> set of relevant chunk ids.  runs: retriever -> query -> ranked id list.
GOLD = {
  "q1": {"c1"}, "q2": {"c2"}, "q3": {"c3","c9"}, "q4": {"c4"}, "q5": {"c5"},
}
RUN_A = {  # retriever A: often buries gold mid-list
  "q1": ["c7","c1","c3"], "q2": ["c2","c8","c0"], "q3": ["c1","c9","c3"],
  "q4": ["c0","c6","c4"], "q5": ["c5","c2","c1"],
}
RUN_B = {  # retriever B: gold higher on average
  "q1": ["c1","c7","c3"], "q2": ["c2","c8","c0"], "q3": ["c3","c9","c1"],
  "q4": ["c4","c0","c6"], "q5": ["c1","c5","c2"],
}

def recall_at_k(run, k):
    hit = 0
    for q, rel in GOLD.items():
        hit += any(cid in rel for cid in run[q][:k])
    return hit / len(GOLD)

def mrr(run):
    total = 0.0
    for q, rel in GOLD.items():
        for i, cid in enumerate(run[q], 1):
            if cid in rel: total += 1.0/i; break
    return total / len(GOLD)

# TODO: implement ndcg_at_k(run, k): DCG = sum over ranked positions of rel/log2(rank+1),
#       with rel=1 if the doc is in gold else 0; normalize by the ideal DCG (all gold on top).

def ndcg_at_k(run, k):
    total = 0.0
    for q, rel in GOLD.items():
        dcg = sum((1.0 if cid in rel else 0.0)/math.log2(i+1)
                  for i, cid in enumerate(run[q][:k], 1))
        ideal_hits = min(len(rel), k)
        idcg = sum(1.0/math.log2(i+1) for i in range(1, ideal_hits+1))
        total += (dcg/idcg) if idcg else 0.0
    return total / len(GOLD)

for name, run in (("A", RUN_A), ("B", RUN_B)):
    print(f"{name}: recall@3={recall_at_k(run,3):.2f}  recall@1={recall_at_k(run,1):.2f} "
          f" MRR={mrr(run):.2f}  nDCG@3={ndcg_at_k(run,3):.2f}")