Free lesson · The AI Evaluation Engineer · Lesson 7
Did Search Find the Right Passage?
In a RAG system, the model can only use what the search step hands it. If the right passage never arrives, no prompt and no model can fix the answer. Your team is about to report a search score from a well-known benchmark. Before they do, you'll measure search yourself, find out what the benchmark is really measuring, and test it with questions in your own words.
How this lesson works
You'll play the search engine on one question, compute search scores by hand, run a full search over 2,067 Wikipedia paragraphs, and then try to break it with your own wording. Plan on about an hour. The script needs only Python, nothing to install.
The data
SQuAD is a reading-comprehension dataset from Stanford. People were shown paragraphs from Wikipedia articles and wrote questions that each paragraph answers. This lesson uses its development set: 2,067 paragraphs from 48 articles and 10,570 questions. For every question we know which paragraph holds the answer, so we can check whether a search finds it.
The search is BM25, the standard keyword ranking behind most search engines and many RAG systems. It scores each paragraph by the question's words it contains, giving more weight to rare words.
Source: Rajpurkar et al., 2016, "SQuAD: 100,000+ Questions for Machine Comprehension of Text." Dataset: SQuAD 1.1, CC-BY-SA 4.0. Paragraphs are Wikipedia text, CC-BY-SA.
Step 1
Be the search engine
A user asks:
Here are four paragraphs from the collection. Which one answers the question?
Predict
Where do you think keyword search ranked the paragraph with the answer, out of 2,067?
The answer is in the paragraph that begins "In 1501, at the age of 19, he entered the University of Erfurt." You probably found it in seconds, because you understood that "entered the University" answers "go to school" and that "he" is Luther.
Keyword search didn't return it at all. After dropping small words like "where" and "did," the question's words are "martin," "luther," "go" and "school." The paragraph contains none of them. It calls him "he" and says "University," not "school." BM25 only scores paragraphs that share a word with the question, so this one never entered the list. Look at what it ranked instead: first, a paragraph about Luther's birth and family, which names him; second, a paragraph about Nikola Tesla, which contains "Martin" (his math teacher, Martin Sekulić) and "school"; third, one that contains "Luther" and the word "go" ("one half was to go to the building of St Peter's"). That's keyword matching: it finds the right words and misses what the question means.
Step 2
Score search by hand
Three numbers describe how well search works:
Top-k rate (often called recall@k): the share of questions whose right paragraph appears in the first k results.
Reciprocal rank for one question: 1 divided by the position of the right paragraph (first place gives 1, third gives 1/3). It's 0 if the paragraph isn't returned.
Mean reciprocal rank (MRR): the average of those, which rewards putting the right paragraph near the top.
Here are ten real questions and where search placed each right paragraph:
The Martin Luther question counts as 0 in every number. It's easy to drop questions like that because they have no rank, and dropping them makes search look better than it is.
Read the BM25 class first; it's about 25 lines and it is the whole search engine. You should see:
right paragraph in the top 1 77%
right paragraph in the top 3 89%
right paragraph in the top 5 92%
right paragraph in the top 10 94%
right paragraph in the top 20 96%
mean reciprocal rank 0.84
not returned at all 0.7%
When the top result is wrong, it comes from the same article 76% of the time.
Explain it
A teammate wants to report "our search finds the right passage 92% of the time." What would you want to know before that sentence goes into a report?
When the top result is wrong, three times out of four it's from the right article. What does that suggest about the kind of mistake search makes, and how you might fix it?
1. How the questions were made. 92% is a real result on this benchmark, but these questions were written by people looking at the paragraph that answers them. That matters more than it seems, and Step 4 measures how much. You'd also want to know the top-k (how many passages go to the model) and whether the questions resemble your users'.
2. Near misses. Search usually lands in the right neighborhood and picks the wrong paragraph from it. That's the typical case for adding a reranker, a second, slower model that reorders the top 20 or so results by reading each one against the question. It can't help when the right paragraph isn't in the list at all, like the Martin Luther one.
Step 4
What the benchmark is measuring
The people who wrote SQuAD's questions were reading the paragraph at the time, so their questions tend to reuse its words. Real users asking your product don't know the wording of the passage they need. The script can split questions by how many of their words also appear in the answering paragraph.
Predict first
For questions that share a third or less of their words with the answering paragraph, search finds it in the top 5 about % of the time.
Run python retrieval_lesson.py overlap:
words shared questions top 1 top 5
a third or less 9% 17% 39%
about half 43% 71% 94%
two thirds or more 49% 93% 100%
Half of the benchmark's questions reuse at least two thirds of their words from the answer, and for those, search is nearly perfect. For the 9% phrased mostly in other words, it finds the right paragraph in the top 5 only 39% of the time. The headline 92% says as much about how the questions were written as about the search engine.
Your turn · try to break it
The ask command runs any question you type and tells you where the first paragraph containing the answer ranks. Try the Doctor Who question from the dataset, then your own wording:
python retrieval_lesson.py ask "What type/genre of TV show is Doctor Who?" "science-fiction"
python retrieval_lesson.py ask "What genre of programme is Doctor Who?" "science-fiction"
Then pick any fact you know is in these articles (the 48 topics include the Super Bowl, Nikola Tesla, the Rhine, the Apollo program and Martin Luther) and ask about it the way a person who didn't know the wording would. Record what happened.
The first Doctor Who question ranks the right paragraph 45th. Change "show" to "programme," the British word the paragraph uses, and it jumps to 1st. One word decided whether the model would see the answer.
Keyword search matches spelling and knows nothing about meaning. The usual fixes are to add semantic search, which compares meanings using embeddings, alongside keyword search; to expand queries with synonyms; and, most important for evaluation, to test with questions written the way your users write them, not the way your documents are written.
Step 5
How many passages should the model get?
Handing the model more passages raises the chance the right one is among them, but every passage costs tokens and adds reading the model may get wrong. SQuAD paragraphs average about 120 words, roughly 160 tokens.
Decide
From Step 3: the right paragraph is in the top 1 for 77% of questions, the top 5 for 92%, and the top 20 for 96%. You answer 100,000 questions a day. Going from 5 passages to 20 adds 15 × 160 = 2,400 input tokens per question. How many more questions a day get the right paragraph, and how many extra tokens does that cost per additional question helped?
More questions helped per day: Extra tokens per additional question helped:
Going from 92% to 96% helps 4% of 100,000 = 4,000 more questions a day. The extra cost is 2,400 tokens × 100,000 = 240 million tokens a day, or 60,000 tokens per additional question helped.
Whether that's worth it depends on your token price and what a wrong answer costs. Two better options usually come first: a reranker, which moves the near misses from Step 3 into the top 5 without sending more text; and fixing the vocabulary gap from Step 4, which extra passages only partly close: for questions that share few words with their paragraph, 36% aren't in the top 20 either.
Step 6
Test search for a real product, no hints
Your company is building an assistant that answers employees' questions from the internal wiki, about 5,000 pages. The engineer who built search wrote 200 test questions by opening wiki pages and writing a question each page answers. Search finds the right page in the top 5 for 95% of them, and they want to ship.
Your plan
What's wrong with the test, and how would you build a better one? Be specific about where the questions come from, what you'd measure and what you'd report.
What's wrong: it's the SQuAD problem. Questions written while looking at the page reuse the page's words, and you saw that search is nearly perfect on those and poor on questions phrased differently. The 95% measures whether search can find a page using the page's own vocabulary. Employees won't use it: they'll write "how do I get reimbursed for a flight" when the page is titled "Travel expense policy."
A better test:
Take questions from real users: the existing wiki search log, help-desk tickets, chat messages. Have someone who knows the wiki mark which pages answer each one, and keep the ones with no answer page as a separate group.
Report by slice: overall top-5, plus top-5 for questions that share few words with their page, the slice where Step 4 showed search breaks.
Count every question, including ones whose page isn't returned at all, as you did in Step 2.
Compare fixes on the same set: keyword search, semantic search, both combined, with and without a reranker. Then choose how many passages to send using a cost calculation like Step 5.
Step 7
How your judgment did
What to take away
Measure search separately from answers: if the right passage doesn't arrive, nothing downstream can fix it. Count every question, including ones with no result. Ask how the test questions were written, because questions written from the answer text flatter keyword search. Split results by how much a question shares with its answer. Look at near misses before sending more passages, and work out what extra passages cost per question helped.
Data: SQuAD 1.1, Rajpurkar, P. et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP. CC-BY-SA 4.0. BM25: Robertson, S. and Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond.