Retrieval Without the Math
You'll hear your team say "recall@5" and "MRR." You don't need the formulas — you need to know what question each one answers, so you can tell a real quality report from a comforting one.
There is really one question underneath all the jargon: when a user asks something, did the system actually pull up the passage that contains the answer? "Recall@5" is just the fraction of questions where the answer-bearing passage made it into the top five results. If recall@5 is 0.64, then for roughly one in three questions the answer wasn't even in front of the model — so no amount of clever wording could have saved the response. "MRR" (mean reciprocal rank) is a companion number that captures how high the right passage ranked, since being at position 1 is much better than position 5.
The leadership takeaway is small and powerful: ask your team for one number — of real questions, what fraction had the answer actually retrieved? A team that can answer that is measuring the thing that matters. A team that can't is flying on vibes.
The prompt that can't win
An engineer says wrong answers will be fixed by "improving the prompt." You ask one question that tells you whether that's even possible. What is it?
Ask: "For these wrong answers, was the correct passage actually retrieved?" If it wasn't, the answer was physically absent from what the model saw — and no prompt can conjure information that isn't there. That's a retrieval problem, fixed upstream (better documents, better search). Only if the right passage was in front of the model and it still answered wrong is prompting the right lever. This one question stops your team from spending a sprint polishing prompts that can't possibly work.
CoverAll: one in three answers built on nothing
CoverAll, an insurance assistant, looked fine in spot checks — the answers read fluently and confidently. When the team finally measured retrieval on 250 real questions, recall@5 was 0.64: for over a third of questions, the passage that actually contained the answer never made it into the results the model saw. The fluent answers for those questions were the model improvising in the shape of an answer.
The revelation reframed their whole roadmap. They'd been planning prompt experiments and a model upgrade; the number said the bottleneck was upstream — the system simply wasn't finding the right passages. They redirected effort to search quality (the subject of the next chapter), and answer accuracy followed retrieval up. The lesson for their VP: ask for the retrieval number first; it tells you where the problem even lives.
Brightline adopts one house rule for every RAG status update: lead with retrieval. "Of real employee questions this week, what fraction had the answer actually retrieved?" The first honest answer — 0.71 — is uncomfortable, and exactly the number they needed to start improving instead of guessing.
Quiz · Chapter 4
- "Recall@5 is 0.64" means:
- If the right passage was never retrieved, a better prompt will:
- The one number a leader should ask for is:
- A fluent, confident answer with low retrieval is: