Tools of the Trade & Interview Prep
You now have the mental models and the measurement discipline. This chapter maps them onto the real tools you'll name in an interview, then rehearses the questions — with the senior answers — that separate someone who has shipped search from someone who has read about it.
The stack, by stage
| Stage | Tools you should be able to name | What to say about choosing |
|---|---|---|
| Chunking / ingest | LangChain / LlamaIndex splitters, Unstructured, custom structure-aware splitters | Prefer structure-aware; measure recall vs. size on your corpus (Ch. 3) |
| Embeddings | Domain/general embedding models via the provider SDK; open models for on-prem | Pick for your domain; match query/doc embedding conventions (Ch. 4) |
| Vector store / ANN | pgvector, Pinecone, Qdrant, Weaviate, Milvus (HNSW under the hood) | pgvector if you already run Postgres; managed if you need scale/ops off your plate |
| Lexical | Elasticsearch / OpenSearch BM25, or rank-bm25 for small sets | You need it for hybrid — exact-term recall (Ch. 5) |
| Reranking | Cross-encoder rerankers (hosted or open) | Retrieve wide, rerank narrow; measure precision@k lift (Ch. 6) |
| Generation | Anthropic SDK, claude-opus-4-8; structured outputs for citations | Ground explicitly; require per-claim citations (Ch. 9) |
| Eval / monitoring | Ragas-style metrics, custom recall@k harness, LLM-as-judge for faithfulness | Component evals + online deltas; own the harness (Ch. 8, 10) |
"I treat RAG as a pipeline of measurable stages. I localize every failure to a stage with component evals — recall@k for retrieval, faithfulness for generation — and I only add complexity (hybrid, reranking, agentic loops) when the harness proves it moves the number." That sentence signals you build search the way senior engineers do: empirically.
Interview simulation — the questions and the senior answers
Q1. "Our RAG bot gives wrong answers. Walk me through debugging it."
Weak answer: "I'd improve the prompt / try a bigger model." Senior answer: "First I localize: for the failing queries, is the gold chunk in the retrieved set? If no, it's a retrieval failure — I look at chunking, hybrid, and recall@k. If yes, it's a generation failure — I look at grounding instructions and citations. I don't touch anything until component evals tell me which stage is bleeding." (Ch. 2, 8.)
Q2. "When would you use keyword search over vector search?"
Senior answer: "Exact identifiers, error codes, part numbers, rare jargon — anything where the token itself is the signal. Dense search blurs 'XR-500' toward 'XR-400.' In practice I don't choose; I run both and fuse with RRF, because they miss different queries and the union beats either alone." (Ch. 5.)
Q3. "Retrieval recall@50 is 0.94 but users still get bad answers. What's wrong?"
Senior answer: "The answer's in the candidate set but not near the top, so synthesis never sees it — a ranking problem. I'd add a cross-encoder reranker and measure recall@3 and MRR before/after; I'd expect recall@50 unchanged but recall@3 to jump. If gold chunks are present in top-3 and answers are still wrong, then it's a generation/faithfulness problem, not retrieval." (Ch. 6, 8, 9.)
Q4. "How do you know a change actually improved the system?"
Senior answer: "A fixed, stratified gold set of real queries with labeled relevant chunks. Every change is a before/after on recall@k, MRR, and faithfulness — with confidence intervals so I don't ship noise. In production I watch online deltas: citation-coverage, answer-rate, re-ask rate. I never ship on 'it feels better.'" (Ch. 8, 10.)
Q5. "The model hallucinates even with retrieval. Fixes?"
Senior answer: "Ground explicitly — 'answer only from context, else say you don't know' — and require per-claim citations so unfaithfulness is measurable. I score faithfulness by checking each claim against its cited chunk, and I treat honest refusals as success, not failure, so I don't train the system to guess. If a specific fact is missed mid-context, I reorder top chunks to the edges for lost-in-the-middle." (Ch. 9.)
How Priya got the offer
Priya had no "RAG engineer" title on her résumé. What she had was one repo: a hybrid pipeline over 2,000 public documentation pages — section-aware chunking, dense + BM25 fused with RRF, a cross-encoder reranker, and cite-or-abstain generation — wrapped in an eval harness that reported recall@k and faithfulness on a 500-query gold set. Her write-up led with a finding: dense-only retrieval scored 0.55 on exact-identifier queries; after adding lexical + RRF it hit 0.90, and she showed the before/after by slice.
In the interview she didn't recite definitions. Asked "how would you evaluate this?", she opened her own dashboard. Asked "the bot gave a wrong answer — what first?", she described localizing to a stage from her own logs. The measured artifact was the interview. One real, instrumented pipeline beat every candidate who could only talk about RAG in the abstract.
Acme's finished stack — section-aware chunks, dense + BM25 + RRF, cross-encoder rerank, cite-or-abstain generation, a sliced gold set with online monitoring — is exactly the architecture you can now draw on a whiteboard and defend, stage by stage, in an interview.
Quiz · Chapter 12 — reasoning, not recall
- An interviewer asks how you'd debug wrong answers. The strongest opening move is:
- You'd reach for BM25 lexical search specifically when:
- recall@50 is high but answers are bad and gold chunks sit at rank ~30. The right fix and its expected signature:
- What single practice most signals senior RAG judgment?
- Treating an honest "I don't know" (when context lacks the answer) as a success in eval is correct because: