Why Search Came Back
When large language models arrived, plenty of people predicted the end of search. The opposite happened. Retrieval moved inside the model's loop, and deciding what goes into the prompt became one of the most important jobs in AI engineering.
Clarity Legal's $1.30 question
Clarity Legal built a contract-review assistant without any retrieval. Every prompt included all 900 standard clauses, about 260,000 tokens, and the model was left to find the relevant one. The demo went well. In production, each question cost about $1.30, or roughly $78,000 a month at their volume. Accuracy also suffered: when the right clause sat in the middle of that wall of text, the correct-answer rate fell from 94% to 61%.
They rebuilt it to retrieve the three relevant clauses, around 800 tokens, for each question. With the same model and the same questions, cost fell to about $0.004 per question and accuracy on mid-document clauses went back up to 91%.
The question with no honest answer
Ask a strong model about something it never saw in training:
What is the rate limit on the Acme Vault API's /export endpoint?It has never read the Acme Vault docs. It can say "I don't know," or it can produce something shaped like a rate limit, such as "100 requests per minute," because answers of that shape are everywhere in its training data. Models usually do the second. You can't prompt that away; it's what a next-word predictor does when the real answer isn't in its weights. What works is putting the answer in front of the model when the question arrives. Retrieve the relevant text, give it to the model and ask it to answer from that text. That's retrieval-augmented generation, and building the retrieval half well is the RAG engineer's job.
Why not put everything in the context window?
Context windows can now hold a million tokens, so it's tempting to skip retrieval and paste the whole knowledge base into every prompt. Two numbers argue against it.
Cost. Take 800 documents at about 300 tokens each, roughly 240,000 tokens. At around $5 per million input tokens, sending all of it costs about $1.20 per question. Sending the two documents that matter, about 600 tokens, costs about $0.003. That's a 400× difference on every query. At 100,000 queries a month, it's the difference between a $300 bill and a $120,000 one.
Accuracy. Models don't pay equal attention across a long context. Facts in the middle of a long prompt are recalled much worse than facts near the start or end, an effect known as "lost in the middle." A fact the model gets right 95% of the time near the top can drop into the 60s when it's buried among 200 documents. So sending everything can cost 400× more and still give worse answers than sending the two right paragraphs.
Long context didn't replace retrieval. Retrieval became the step that decides what goes into the context, and the goal is to send the model the right few hundred tokens.
What the RAG engineer owns
A RAG system is a pipeline. Documents are split into chunks (Chapter 3), indexed as vectors, keywords or both (Chapters 4 and 5), retrieved for each query, reranked so the best ones rise to the top (Chapter 6), and passed to the model to write a grounded, cited answer (Chapter 9). When the answer is bad, the model is rarely the whole story. Usually one stage upstream let the right passage slip away.
That gives you two kinds of failure to tell apart. In a retrieval failure, the passage with the answer never reached the model, and you fix it upstream in chunking or search. In a generation failure, the passage was there and the model ignored or contradicted it, and you fix the prompt and grounding. Measuring them separately (Chapters 8 and 10) keeps you from tuning the wrong stage.
Measure what retrieval is worth
Build a tiny RAG loop over docs for a made-up product, so the model can't know the answers. Ask six questions twice, once straight to the model and once with retrieved context, and count how many each way gets right. Leave USE_REAL_API = False to run free on the mock. You want two numbers, like "no context 0/6, grounded 6/6," and a sense of which misses were retrieval and which were generation.
Starter code
import re, anthropic
USE_REAL_API = False # flip to True if you have a key; mock runs free
CORPUS = [
"The Acme Vault API /export endpoint is rate limited to 20 requests per minute per API key.",
"Acme Vault stores secrets encrypted with AES-256; keys rotate every 90 days.",
"The /export endpoint returns data as newline-delimited JSON, not a single array.",
"Acme Vault's free tier allows 3 projects; the Team tier allows 50.",
"Support for the legacy /dump endpoint ended in v4; use /export instead.",
"Acme Vault webhooks retry failed deliveries up to 5 times with exponential backoff.",
]
QA = [ # (question, correct-answer substring the grader checks for)
("What is the rate limit on the /export endpoint?", "20"),
("What format does /export return?", "newline-delimited"),
("How many projects does the free tier allow?", "3"),
("How often do Acme Vault encryption keys rotate?", "90 days"),
("How many times are failed webhooks retried?", "5"),
("What replaced the legacy /dump endpoint?", "/export"),
]
def retrieve(query, k=2):
q = set(re.findall(r"[a-z0-9]+", query.lower()))
scored = [(len(q & set(re.findall(r"[a-z0-9]+", doc.lower()))), doc) for doc in CORPUS]
scored.sort(reverse=True)
return [doc for score, doc in scored[:k]]
def answer(question, use_context):
context = "\n".join(retrieve(question)) if use_context else ""
prompt = (f"Answer ONLY from the context. If it's not there, say 'unknown'.\n\n"
f"Context:\n{context}\n\nQuestion: {question}") if use_context else \
(f"Answer concisely.\n\nQuestion: {question}")
if not USE_REAL_API:
return _mock(question, use_context)
client = anthropic.Anthropic()
r = client.messages.create(model="claude-opus-4-8", max_tokens=120,
messages=[{"role": "user", "content": prompt}])
return r.content[0].text
# TODO (Step 3): loop QA, call answer() both ways, count how many contain the
# expected substring (case-insensitive), print "no-context X/6, grounded Y/6".The scoring loop turns "grounding helps" into a number:
def score(use_context):
hits = sum(exp.lower() in answer(q, use_context).lower() for q, exp in QA)
return hits
print(f"no-context {score(False)}/{len(QA)}, grounded {score(True)}/{len(QA)}")
# Recorded mock so the numbers are real without a key:
_MOCK = {
# (question, use_context) -> model output
("What is the rate limit on the /export endpoint?", False): "Typically 100 requests per minute.",
("What is the rate limit on the /export endpoint?", True): "20 requests per minute per API key.",
("What format does /export return?", False): "Usually a JSON array of objects.",
("What format does /export return?", True): "Newline-delimited JSON.",
("How many projects does the free tier allow?", False): "Most free tiers allow around 5 projects.",
("How many projects does the free tier allow?", True): "3 projects.",
("How often do Acme Vault encryption keys rotate?", False): "Commonly every 30 days.",
("How often do Acme Vault encryption keys rotate?", True): "Every 90 days.",
("How many times are failed webhooks retried?", False): "Often 3 retries.",
("How many times are failed webhooks retried?", True): "Up to 5 times with backoff.",
("What replaced the legacy /dump endpoint?", False): "Unknown without documentation.",
("What replaced the legacy /dump endpoint?", True): "The /export endpoint.",
}
def _mock(q, use_context): return _MOCK[(q, use_context)]You should see 0/6 without context and 6/6 with it. The ungrounded answers are fluent and wrong, like "100 requests per minute" and "every 30 days." With two retrieved lines, all six are right. The last question, about what replaced /dump, is worth a look: without context the model said it didn't know, which is honest, and with context it answered correctly. Now set k=1 and run it again. Some questions whose answer was in the second-ranked document will fail. Those are retrieval failures, because the right text never reached the model.
For extra practice, add a distractor document that shares words with a question but doesn't answer it, and watch the word-overlap retriever rank it highly. Chapters 4 and 5 are about fixing exactly that.
Acme's first "ask the docs" bot pasted the whole 180-page Acme Vault manual into every prompt. It cost about $0.90 per question and kept missing facts from the middle of the manual. That's why Acme is now building real retrieval, and we'll follow the build to the end of the book.
You don't need to memorize token prices, but you should be able to do the rough math: corpus size times price per token, compared with a few paragraphs times the same price. Interviewers like candidates who can put numbers on a design choice in their head.
"Lost in the middle" is also worth reading about in its original paper. It's short, it explains a behavior you'll see in every RAG system, and mentioning it by name tells an interviewer you've gone past blog posts.
A "chat with your docs" demo is three questions someone already knew would work. Your customers will ask thousands you never tried. Before approving a launch, ask how many real questions the system was measured on, and whether it can tell a failure to find the answer from a failure to use it. Those need different fixes, and a team that can't separate them is guessing.
Ask about cost per question, too. If the answer is "we send everything, the context window is huge," expect a large bill and worse answers than you'd get from proper retrieval.
Quiz · Chapter 1
- Your RAG app gives a wrong answer about a document you definitely ingested, and it cites nothing from it. What do you check first?
- A teammate wants to skip retrieval and paste all 800 docs into every prompt. What's the strongest objection?
- Without retrieval, the model says the /export rate limit is "100 requests per minute." What's going on?
- You lower k from 2 to 1 and three correct answers start failing. What did that show?
- Why separate retrieval failures from generation failures?