Chapter 2

Anatomy of a RAG Pipeline

"The answer was wrong" describes a symptom. A RAG system is five stages in a row, and a bad answer almost always traces back to one of them. This chapter gives you a way to find which.

Case study

MediChat blames the model

MediChat answers clinicians' questions from a library of care guidelines. A reviewer caught it giving a wrong dosing range, and the team's first idea was to try a bigger model. First they checked whether the right guideline passage had been retrieved. It had ranked 14th, below their cutoff of 8, so the model never saw it.

They then tagged 50 flagged answers by the stage that failed: 19 retrieval failures, 22 ranking failures where the passage was retrieved but buried, and 9 generation failures. Their next sprint went to chunking and search instead of the prompt they'd been about to rewrite.

The pipeline, end to end

OFFLINE (build the index, once per document)
  documents ──▶ [1] chunk ──▶ [2] embed / index ──▶ vector + lexical store

ONLINE (per query)
  query ──▶ [3] retrieve ──▶ [4] rerank ──▶ [5] synthesize ──▶ grounded answer
                 (top 50)      (top 5)         (LLM + context)

Every stage can drop the information you need, and once it's gone, nothing later can bring it back. If chunking split a rate-limit table in two, retrieval can't put it back together. If the answer isn't among the 50 retrieved candidates, reranking can't find it. If it's at rank 40 and only the top 5 go to the model, the model never sees it. Information only gets lost as you move from left to right.

The five stages and how each fails

Stage	Job	Typical failure	Chapter
1 · Chunk	Split documents into retrievable pieces	An answer split across two chunks; a chunk mixing three topics	3
2 · Embed / index	Make chunks searchable	An embedding model that doesn't fit the domain; an index missing new docs	4–5
3 · Retrieve	Pull candidate chunks for a query	The right chunk isn't in the top k at all	4–5
4 · Rerank	Put the best candidates first	The right chunk is retrieved at rank 30 and cut before the model	6
5 · Synthesize	Write a grounded, cited answer	The chunk is there but the model ignores or contradicts it	9

The fixes differ a lot. A miss at stage 3 calls for better chunking or hybrid search, which is a data problem. A miss at stage 5 with the right chunk present is a prompt problem.

The check that splits every failure

For each failing query, ask whether the chunk with the answer (the "gold" chunk) reached the model. If it didn't, the problem is somewhere in stages 1 to 4. If it did and the answer is still wrong, it's stage 5. The share of queries whose gold chunk is retrieved is called retrieval recall, and it's the first number to look at.

That's why careful teams track retrieval separately from final answer quality. A system can hit 95% answer accuracy with 99% recall, which means generation is doing its job. It can also hit 95% with 70% recall, which means the model is covering for weak retrieval, and one shift in the documents or the questions could drop it sharply. The end-to-end number looks the same either way.

Try it · ~20 minFree · no API key

Assign every failure to a stage

You get recorded retrieval results and answers for a few queries, some of them wrong. Write a function that labels each failure "retrieval" or "generation" by checking whether the gold chunk was in the retrieved set, then print the counts and the recall. Decide which stage to fix first.

Starter code

# Recorded run: each row is one query with what actually happened.
# gold = id of the chunk that contains the answer
# retrieved = ids the retriever returned (top-k)
# correct = did the final answer match the gold answer?
RUN = [
    {"q": "export rate limit?",      "gold": "c12", "retrieved": ["c12","c04"], "correct": True},
    {"q": "key rotation period?",    "gold": "c07", "retrieved": ["c01","c04"], "correct": False},
    {"q": "free tier project cap?",  "gold": "c19", "retrieved": ["c19","c02"], "correct": True},
    {"q": "webhook retry count?",    "gold": "c33", "retrieved": ["c31","c33"], "correct": False},
    {"q": "what replaced /dump?",    "gold": "c22", "retrieved": ["c05","c08"], "correct": False},
]

# TODO: for each row, classify:
#   gold not in retrieved  -> "retrieval"   (answer never reached the model)
#   gold in retrieved but not correct -> "generation" (model had it, blew it)
#   correct -> "ok"
# Then print counts per class and retrieval recall = (# rows with gold in retrieved)/len(RUN)

Worked solution

from collections import Counter

def classify(row):
    hit = row["gold"] in row["retrieved"]
    if row["correct"]:      return "ok"
    return "generation" if hit else "retrieval"

labels = [classify(r) for r in RUN]
counts = Counter(labels)
recall = sum(r["gold"] in r["retrieved"] for r in RUN) / len(RUN)

print(counts)                       # Counter({'ok': 2, 'retrieval': 2, 'generation': 1})
print(f"retrieval recall: {recall:.2f}")   # 0.60

You should see two retrieval failures, one generation failure and a recall of 0.60. Retrieval is the bigger problem: 40% of gold chunks never reach the model. Improving the prompt could at best fix the one generation failure. The webhook question is different from the other two, since its gold chunk was retrieved and the answer was still wrong. That's the one where prompt work will pay off, and without this check you wouldn't know.

To go further, add a third label for chunks that were retrieved but ranked below the cutoff you pass to the model. Those look like retrieval failures but are really ranking failures, which is what Chapter 6 fixes.

Running case · Acme Vault

Acme starts tagging every thumbs-down as retrieval, ranking or generation. The first week's 50 complaints break down 31, 12 and 7, so Acme fixes chunking before touching a single prompt.

If you're starting out

The five-stage diagram is worth drawing from memory. When an interviewer asks how RAG works, drawing it and naming how each stage fails is a clear, organized answer, and it leads straight into the debugging questions they'll ask next.

"Gold chunk" and "recall" come from information retrieval. If they're new to you, an introductory IR textbook chapter on evaluation will cover them in an evening.

If you run the team

You don't need to build the pipeline, but you should be able to recognize a team that understands it. Ask them to draw it. Then ask, for a recent bad answer, which stage it came from. A team that can answer improves steadily; a team that can't tends to rewrite prompts and hope.

Also ask whether releases are checked against a fixed set of questions and whether someone watches quality on live traffic. You need both: one catches problems before launch and the other catches the ones nobody anticipated.