Chapter 8

Evaluating Retrieval

Chunk size, hybrid search, reranking and query rewriting are all settings you can change. Without measurement, changing them is guesswork. This chapter builds the evaluation harness that turns "I think hybrid helped" into "recall@10 went from 0.71 to 0.83 on our 80 labeled questions."

Case study

InsureBot's 0.90 hid a 0.62

InsureBot was ready to ship with recall@5 of 0.90 on a 300-question gold set. Before launch, they split the number by question type. FAQ-style questions scored 0.98. Questions about newly added policy documents scored 0.62, and those made up about a quarter of real traffic.

The gold set made this visible: real questions, each paired with the chunk that answers it, tagged by type and large enough that a 70-question slice was still stable. The slices told them where to spend the next sprint.

You can't improve retrieval you don't measure, and you can't measure it without a gold set: realistic queries, each labeled with the chunk or chunks that answer it. Building one is unglamorous and very valuable. Fifty to a hundred labeled queries is enough to make most decisions on evidence.

Four metrics to know

Metric	Question it answers	Use it for
Recall@k	Is the gold chunk in the top k?	Retrieval: did the answer make it into the candidates?
MRR	How high is the first relevant chunk?	Reranking and single-answer questions
nDCG@k	Are several relevant chunks ranked high, weighted by position?	Questions with more than one relevant chunk
Precision@k	What share of the top k is relevant?	How much noise reaches the model

For the retrieval stage, recall@k matters most, because it measures whether the answer reaches the model at all, and nothing later can fix a miss. Tune retrieval for recall at a k you can afford to rerank, such as recall@50, then let reranking turn that into precision in the top 3.

Measure the stages separately

A single "is the final answer good?" score tells you something is wrong without telling you where. Measure retrieval with recall@k and generation with faithfulness (Chapter 9) on the same gold set, and you'll know which stage to work on.

Building the gold set

You don't have to label a hundred queries from scratch. Take real queries from your logs, retrieve candidates, and have people mark which chunks are relevant. A strong model can do a first pass for people to check. Spread the queries across your real query types, such as ID lookups, how-to questions and multi-part questions. Fifty queries that cover your real mix are more useful than five hundred of one kind.

Try it · ~30 minFree · no API key

Build the retrieval evaluation harness

Implement recall@k, MRR and nDCG from scratch over a small gold set, use them to compare two retrievers, and pick a winner from the numbers. This harness is the thing you'll carry to every RAG system you work on.

Starter code

import math

# gold: query -> set of relevant chunk ids.  runs: retriever -> query -> ranked id list.
GOLD = {
  "q1": {"c1"}, "q2": {"c2"}, "q3": {"c3","c9"}, "q4": {"c4"}, "q5": {"c5"},
}
RUN_A = {  # retriever A: often buries gold mid-list
  "q1": ["c7","c1","c3"], "q2": ["c2","c8","c0"], "q3": ["c1","c9","c3"],
  "q4": ["c0","c6","c4"], "q5": ["c5","c2","c1"],
}
RUN_B = {  # retriever B: gold higher on average
  "q1": ["c1","c7","c3"], "q2": ["c2","c8","c0"], "q3": ["c3","c9","c1"],
  "q4": ["c4","c0","c6"], "q5": ["c1","c5","c2"],
}

def recall_at_k(run, k):
    hit = 0
    for q, rel in GOLD.items():
        hit += any(cid in rel for cid in run[q][:k])
    return hit / len(GOLD)

def mrr(run):
    total = 0.0
    for q, rel in GOLD.items():
        for i, cid in enumerate(run[q], 1):
            if cid in rel: total += 1.0/i; break
    return total / len(GOLD)

# TODO: implement ndcg_at_k(run, k): DCG = sum over ranked positions of rel/log2(rank+1),
#       with rel=1 if the doc is in gold else 0; normalize by the ideal DCG (all gold on top).

Worked solution

def ndcg_at_k(run, k):
    total = 0.0
    for q, rel in GOLD.items():
        dcg = sum((1.0 if cid in rel else 0.0)/math.log2(i+1)
                  for i, cid in enumerate(run[q][:k], 1))
        ideal_hits = min(len(rel), k)
        idcg = sum(1.0/math.log2(i+1) for i in range(1, ideal_hits+1))
        total += (dcg/idcg) if idcg else 0.0
    return total / len(GOLD)

for name, run in (("A", RUN_A), ("B", RUN_B)):
    print(f"{name}: recall@3={recall_at_k(run,3):.2f}  recall@1={recall_at_k(run,1):.2f} "
          f" MRR={mrr(run):.2f}  nDCG@3={ndcg_at_k(run,3):.2f}")

A and B tie on recall@3, since both get the gold document somewhere into the top 3. B wins clearly on recall@1, MRR and nDCG@3, because it puts the gold document higher. Recall@3 alone would have called them equal. With this harness, every change becomes a before-and-after number on a fixed set, and a change ships when its number moves, for example recall@10 up 0.06 and MRR up 0.11.

To go further, add graded relevance (2 for the main answer, 1 for partly relevant chunks) to nDCG, and bootstrap confidence intervals over the queries so you can tell whether a 0.02 gain is real.

Running case · Acme Vault

Acme builds a 300-question gold set tagged as how-to, reference or error-code. Recall@5 comes back at 0.95, 0.91 and 0.64, which points the next sprint at error-code retrieval.

If you're starting out

Knowing these four metrics, and when to use each, covers a large share of RAG interview questions. Practice computing each one by hand on a tiny example. Interviewers often ask you to work through one on a whiteboard.

If you only build one portfolio piece for RAG, make it an evaluation harness with a small, honestly labeled gold set. It shows the habit employers are looking for more clearly than a flashy demo.

If you run the team

A single quality number is comforting and often misleading. Ask for results split by product, question type and document age, with the number of questions in each. An overall 90% can hide a 60% slice that produces most of your complaints, and each customer only ever sees their own slice.

Treat the gold set as an asset with an owner. It should grow as real complaints are turned into new test questions. A test set nobody has touched in a year describes last year's product.

Back Continue