Why Search Came Back

Everyone said large language models would kill search. The opposite happened: retrieval became load-bearing infrastructure inside the model's loop. Understanding exactly why — and what breaks when you skip it — is the whole foundation of the job.

Here is the situation that creates the entire field. You ask a strong model a question about something it was never trained on — your company's refund policy, a document uploaded five minutes ago, an API released last week:

The question that has no honest answer

What is the rate limit on the Acme Vault API's /export endpoint?

The model has never seen the Acme Vault docs. It has two options: say "I don't know," or produce something that sounds like an API rate limit — "100 requests per minute," maybe "1,000 per hour" — because that shape of answer is everywhere in its training data. Models overwhelmingly pick the second. That's not a bug you can prompt away; it's what a next-token predictor does when the real answer isn't in its weights. The fix is not a better model. The fix is to put the answer in front of it at question time. That move — retrieve the relevant text, hand it to the model, ask it to answer from that text — is Retrieval-Augmented Generation, and building the retrieval half of it well is what an AI Search / RAG engineer does.

"Just put everything in the context window" — why that isn't the answer

Context windows are now enormous — a million tokens. So the tempting shortcut is: skip retrieval, dump the entire knowledge base into every prompt, let the model sort it out. Two hard numbers kill that idea.

Cost. Say your corpus is 800 documents averaging 300 tokens — about 240,000 tokens. At Opus input pricing (~$5 per million tokens), stuffing the whole corpus into every query costs about $1.20 per question. Retrieve the 2 documents that actually matter — roughly 600 tokens — and the same question costs about $0.003. That is a ~400× difference, per query, forever. At 100,000 queries a month it's the difference between a $300 bill and a $120,000 bill.

Accuracy. Bigger context isn't free accuracy, either. Models attend unevenly across a long context — facts placed in the middle of a long prompt are recalled far worse than facts at the start or end. This is the well-documented "lost in the middle" effect: take a fact the model answers correctly 95% of the time when it's near the top of the context, bury it in the middle of 200 documents, and accuracy can fall to the 60s. So dumping everything in doesn't just cost 400× more — it can actually lower the answer rate versus handing the model the two right paragraphs. Retrieval is not a cost hack you tolerate; it is how you make the answer better.

What the RAG engineer actually owns

A RAG system is a pipeline, and "it gave a bad answer" almost never means "the model is bad." It usually means a specific stage upstream failed. The stages:

1. Ingest & chunk — split documents into retrievable units (Ch. 3).
2. Index — build the structures you'll search: vectors, a lexical index, or both (Ch. 4–5).
3. Retrieve — given a query, pull candidate chunks (Ch. 4–5).
4. Rerank — reorder candidates so the best ones land at the top (Ch. 6).
5. Synthesize — hand the top chunks to the model and generate a grounded, cited answer (Ch. 9).

The single most important diagnostic skill in the job is knowing which stage failed. If the right document was never retrieved, no amount of prompt engineering on the generation step will save you — the answer physically isn't in the context. If the right document was retrieved but sat at rank 40, reranking is your problem, not retrieval. The discipline is separating these:

The smallest RAG system that teaches the whole idea

You don't need a vector database to feel the effect. A five-line retriever that just counts shared words is enough to turn a confident hallucination into a correct, grounded answer. You'll build exactly that in the lab, and measure the swing.

Starter code

import re, anthropic

USE_REAL_API = False          # flip to True if you have a key; mock runs free

CORPUS = [
    "The Acme Vault API /export endpoint is rate limited to 20 requests per minute per API key.",
    "Acme Vault stores secrets encrypted with AES-256; keys rotate every 90 days.",
    "The /export endpoint returns data as newline-delimited JSON, not a single array.",
    "Acme Vault's free tier allows 3 projects; the Team tier allows 50.",
    "Support for the legacy /dump endpoint ended in v4; use /export instead.",
    "Acme Vault webhooks retry failed deliveries up to 5 times with exponential backoff.",
]

QA = [   # (question, correct-answer substring the grader checks for)
    ("What is the rate limit on the /export endpoint?",        "20"),
    ("What format does /export return?",                        "newline-delimited"),
    ("How many projects does the free tier allow?",             "3"),
    ("How often do Acme Vault encryption keys rotate?",         "90 days"),
    ("How many times are failed webhooks retried?",             "5"),
    ("What replaced the legacy /dump endpoint?",                "/export"),
]

def retrieve(query, k=2):
    q = set(re.findall(r"[a-z0-9]+", query.lower()))
    scored = [(len(q & set(re.findall(r"[a-z0-9]+", doc.lower()))), doc) for doc in CORPUS]
    scored.sort(reverse=True)
    return [doc for score, doc in scored[:k]]

def answer(question, use_context):
    context = "\n".join(retrieve(question)) if use_context else ""
    prompt = (f"Answer ONLY from the context. If it's not there, say 'unknown'.\n\n"
              f"Context:\n{context}\n\nQuestion: {question}") if use_context else \
             (f"Answer concisely.\n\nQuestion: {question}")
    if not USE_REAL_API:
        return _mock(question, use_context)
    client = anthropic.Anthropic()
    r = client.messages.create(model="claude-opus-4-8", max_tokens=120,
                               messages=[{"role": "user", "content": prompt}])
    return r.content[0].text

# TODO (Step 3): loop QA, call answer() both ways, count how many contain the
# expected substring (case-insensitive), print "no-context X/6, grounded Y/6".

def score(use_context):
    hits = sum(exp.lower() in answer(q, use_context).lower() for q, exp in QA)
    return hits

print(f"no-context {score(False)}/{len(QA)}, grounded {score(True)}/{len(QA)}")

# Recorded mock so the numbers are real without a key:
_MOCK = {
  # (question, use_context) -> model output
  ("What is the rate limit on the /export endpoint?", False): "Typically 100 requests per minute.",
  ("What is the rate limit on the /export endpoint?", True):  "20 requests per minute per API key.",
  ("What format does /export return?", False): "Usually a JSON array of objects.",
  ("What format does /export return?", True):  "Newline-delimited JSON.",
  ("How many projects does the free tier allow?", False): "Most free tiers allow around 5 projects.",
  ("How many projects does the free tier allow?", True):  "3 projects.",
  ("How often do Acme Vault encryption keys rotate?", False): "Commonly every 30 days.",
  ("How often do Acme Vault encryption keys rotate?", True):  "Every 90 days.",
  ("How many times are failed webhooks retried?", False): "Often 3 retries.",
  ("How many times are failed webhooks retried?", True):  "Up to 5 times with backoff.",
  ("What replaced the legacy /dump endpoint?", False): "Unknown without documentation.",
  ("What replaced the legacy /dump endpoint?", True):  "The /export endpoint.",
}
def _mock(q, use_context): return _MOCK[(q, use_context)]