AI Search / RAG Engineer · Chapter 2

Anatomy of a RAG Pipeline

"The answer was wrong" is a symptom, not a diagnosis. A RAG system is five stages in a line, and a bad answer traces to exactly one of them. This chapter gives you the mental model that turns "the bot is dumb" into "stage 3 dropped the chunk" — the difference between a search engineer and a prompt-tweaker.

In Chapter 1 you built the smallest possible RAG loop and saw retrieval flip a hallucination into a grounded answer. Now zoom out to the full pipeline, because in production the bug is never "the model." It's one stage, and your entire debugging life is spent localizing which one. Here is the whole system, offline and online halves:

The pipeline, end to end

OFFLINE (build the index, once per document)
  documents ──▶ [1] chunk ──▶ [2] embed / index ──▶ vector + lexical store

ONLINE (per query)
  query ──▶ [3] retrieve ──▶ [4] rerank ──▶ [5] synthesize ──▶ grounded answer
                 (top 50)      (top 5)         (LLM + context)

Each stage can silently drop the right information, and once it's dropped, nothing downstream can recover it. That one-way property is the key to the whole mental model: information can only be lost as you move left to right, never regained. If chunking split a rate-limit table across two chunks, retrieval can't un-split it. If retrieval returned 50 candidates and the answer wasn't among them, reranking can't conjure it. If the answer sat at rank 40 and you only pass the top 5 to the model, synthesis never sees it.

The five stages and how each one fails

Stage	Job	Signature failure	Chapter
1 · Chunk	Split docs into retrievable units	Answer split across two chunks; a chunk mixes three topics so its embedding is muddy	3
2 · Embed / index	Turn chunks into searchable vectors + terms	Wrong embedding model for the domain; stale index missing new docs	4–5
3 · Retrieve	Pull candidate chunks for a query	Answer-bearing chunk not in the top-k at all (recall miss)	4–5
4 · Rerank	Reorder so the best land on top	Right chunk retrieved but at rank 30; truncated before synthesis	6
5 · Synthesize	Generate a grounded, cited answer	Chunk present but model ignores it, contradicts it, or won't cite	9

Notice how differently you'd respond to each. A stage-3 miss means a bigger retriever, hybrid search, or better chunking — a data problem. A stage-5 miss with the right chunk present means a prompt problem. Prescribing the wrong medicine is the single most common waste of time in this work, and it comes from skipping the diagnosis.

The one number that localizes the split: retrieval recall

Before you debug generation, you answer one question: did the answer-bearing chunk even reach the model? That's retrieval recall — the fraction of queries whose gold chunk appears in the retrieved set. It cleanly partitions every failure:

The master diagnostic

For a failing query, check whether the gold chunk is in the retrieved context. Not there → retrieval failure (stages 1–4): fix chunking/retrieval/rerank. There but the answer is still wrong → generation failure (stage 5): fix the prompt, grounding, or citation. You cannot improve what you cannot localize, and this single check localizes everything.

This is why serious teams track retrieval quality separately from end-to-end answer quality. A system can have 95% answer accuracy with 99% retrieval recall (generation is doing its job) or 95% answer accuracy with 70% retrieval recall (generation is quietly compensating, and you're one corpus change away from a cliff). The end-to-end number hides which. You'll build the measurement in Chapter 8; here you build the reflex.

Try it · ~20 min

Instrument the pipeline — attribute every failure to a stage

You'll take a tiny end-to-end RAG run where some answers are wrong, and write a classifier that labels each failure "retrieval" or "generation" by checking whether the gold chunk reached the context. The output is a per-stage failure breakdown — the exact table a senior engineer opens first.

Setup: nothing to install — the lab ships recorded retrieved-sets and answers so it runs offline and deterministic.

Step 1. For each query, check if the gold chunk id is in the retrieved ids.

Step 2. If it isn't and the answer is wrong → retrieval failure. If it is and the answer is still wrong → generation failure.

Step 3. Print counts of each failure class and the retrieval recall.

Your goal: a breakdown like "retrieval failures: 2, generation failures: 1, recall 0.60" — and a one-line call on which stage to fix first.

Starter code

# Recorded run: each row is one query with what actually happened.
# gold = id of the chunk that contains the answer
# retrieved = ids the retriever returned (top-k)
# correct = did the final answer match the gold answer?
RUN = [
    {"q": "export rate limit?",      "gold": "c12", "retrieved": ["c12","c04"], "correct": True},
    {"q": "key rotation period?",    "gold": "c07", "retrieved": ["c01","c04"], "correct": False},
    {"q": "free tier project cap?",  "gold": "c19", "retrieved": ["c19","c02"], "correct": True},
    {"q": "webhook retry count?",    "gold": "c33", "retrieved": ["c31","c33"], "correct": False},
    {"q": "what replaced /dump?",    "gold": "c22", "retrieved": ["c05","c08"], "correct": False},
]

# TODO: for each row, classify:
#   gold not in retrieved  -> "retrieval"   (answer never reached the model)
#   gold in retrieved but not correct -> "generation" (model had it, blew it)
#   correct -> "ok"
# Then print counts per class and retrieval recall = (# rows with gold in retrieved)/len(RUN)

Worked solution

from collections import Counter

def classify(row):
    hit = row["gold"] in row["retrieved"]
    if row["correct"]:      return "ok"
    return "generation" if hit else "retrieval"

labels = [classify(r) for r in RUN]
counts = Counter(labels)
recall = sum(r["gold"] in r["retrieved"] for r in RUN) / len(RUN)

print(counts)                       # Counter({'ok': 2, 'retrieval': 2, 'generation': 1})
print(f"retrieval recall: {recall:.2f}")   # 0.60

What you should see: retrieval: 2, generation: 1, recall 0.60. Read it like a triage nurse: retrieval is the bigger bleed (2 of 3 failures), and recall is only 0.60 — 40% of gold chunks never reach the model. Fixing the generation prompt would, at best, recover the one generation failure while leaving two queries structurally unanswerable. The data tells you to work stages 1–4 first.

The reframe: notice the row "webhook retry count?" — gold c33 was retrieved, yet the answer was wrong. That's the one query where prompt work pays off. Without this classifier you'd never know it was different from the other two failures; you'd "fix the prompt" and watch recall-bound queries keep failing. The breakdown is the whole diagnosis.

Going further: add a fourth class — the gold chunk retrieved but at rank > your synthesis cutoff (say you only pass top-3 but it landed at rank 5). That's a reranking failure masquerading as retrieval, and it's why Chapter 6 exists. Track rank-of-gold, not just presence.

Case study

MediChat: blaming the model for a retrieval bug

MediChat answers clinicians' questions from a library of care guidelines. A reviewer caught it giving a wrong dosing range, and the team's first instinct was "the model is unreliable — let's try a bigger one." Before spending on that, they checked one thing: was the correct guideline chunk even in the retrieved set? It wasn't — it had ranked 14th, below their k=8 cutoff. No prompt or model change could have fixed an answer that was never in the context.

So they instrumented every complaint by failing stage. Over 50 flagged answers: 19 retrieval failures (the chunk never made the candidate set), 22 ranking failures (retrieved but buried), and only 9 generation failures (right chunk, wrong answer). The data pointed their whole next sprint upstream — at chunking and hybrid retrieval — not at the prompt they'd been about to rewrite.

Running case · Acme Vault

Acme adopts the same discipline: every thumbs-down is tagged retrieval / ranking / generation. Week one's 50 complaints break down 31 / 12 / 7 — so Acme fixes chunking before touching a single prompt.