Interview Simulation
Ten questions you will actually be asked, each with what the interviewer is really probing, a strong answer built from this book, and the weak answer that ends the interview. Practice saying these out loud.
RAG interviews rarely ask you to recite definitions. They hand you a vague product — "we want to chat with our support docs" — and watch whether you can turn "make it good" into a pipeline you can diagnose and measure. The single best move in any answer below is to name the stages (ingest & chunk → index → retrieve → rerank → synthesize) and always separate retrieval failures from generation failures. And bring a portfolio: the little pipeline you built is a better answer than any sentence.
1. "Walk me through how you'd build RAG over our internal docs."
Really probing: do you know the pipeline, or just "put docs in a vector DB"?
I'd name the stages: chunk the docs into coherent units with section headings preserved; index them both as dense vectors and a BM25 lexical index; retrieve a candidate set with both and fuse the ranks; rerank the top candidates with a cross-encoder; then generate an answer grounded strictly in the top chunks, with citations, and abstain when the context doesn't support an answer. I'd stand up an eval harness on day one so every change is measured, not guessed.
Weak answer: "Embed everything, drop it in a vector database, and prompt the model." No chunking strategy, no lexical fallback, no reranking, no evaluation — a demo, not a system.
2. "The bot gives a wrong answer about a document we definitely ingested. What do you check first?"
Really probing: can you localize a failure instead of blaming the model?
Before touching the prompt, I check whether the answer-bearing chunk was actually in the retrieved candidate set. If it wasn't, it's a retrieval failure — the answer was physically absent from the context, so no prompt engineering can recover it, and I work chunking / hybrid retrieval / k. If it was retrieved but buried at rank 40, it's a ranking problem for the reranker. Only if the right chunk was in front of the model and it still answered wrong do I treat it as a generation/grounding failure.
Weak answer: "I'd rewrite the prompt" or "try a bigger model." Tuning the wrong stage — the classic tell of someone who's never diagnosed a real pipeline.
3. "Why not skip retrieval and paste all the docs into a 1M-token context?"
Really probing: do you understand cost and the lost-in-the-middle effect?
Two reasons. Cost: stuffing a whole corpus into every query can be hundreds of times more expensive per question than retrieving the two chunks that matter — that compounds forever at scale. And accuracy: models attend unevenly over long contexts, so a fact buried in the middle of hundreds of documents is recalled far worse than one handed over in a two-paragraph context. Retrieval isn't just a cost hack — it often makes the answer better.
Weak answer: "Context windows are big enough now, so retrieval is obsolete." Confidently wrong on both cost and the lost-in-the-middle failure mode.
4. "How do you measure whether your retrieval is any good?"
Really probing: do you evaluate retrieval separately from generation?
I build a gold set of queries paired with the chunk(s) that actually contain the answer, then measure recall@k (did the answer-bearing chunk make the top k?) and MRR (how high did it rank?). That isolates the retriever from the generator, so I know whether a bad answer came from missing context or bad synthesis. I report with a sample size — a score over 20 queries is a rumor.
Weak answer: "I read some outputs and they looked good." No gold set, no recall@k, no separation of stages, no n.
5. "When would you add lexical / BM25 retrieval instead of just embeddings?"
Really probing: do you know the blind spot of dense retrieval?
When exact tokens carry the signal — error codes, API names, SKUs, IDs, rare proper nouns. Embeddings blur those toward near-neighbor forms, so a query for ERR_4021 can miss the one doc that names it. BM25 matches the token directly. I'd run both and fuse with reciprocal rank fusion, because dense and lexical miss different queries, so the union beats either alone.
Weak answer: "Embeddings are strictly better, so lexical is legacy." Misses the exact-match blind spot that hybrid exists to cover.
6. "Reranking is expensive. Justify it."
Really probing: do you know what a cross-encoder buys and costs?
A cross-encoder reads the query and each candidate jointly, so it's far more accurate at ordering than the bi-encoder cosine used for first-stage retrieval — but it must score every candidate, so it's too slow to run over the whole corpus. The pattern is retrieve a broad candidate set cheaply (say top 50), then rerank just those to lift the truly relevant chunk into the top few the model actually reads. It raises recall@3 and MRR while leaving recall@50 fixed — you can measure exactly what it earned.
Weak answer: "It makes results better." No mechanism, no cost model, no measurement.
7. "How do you stop the model from making things up even when retrieval is good?"
Really probing: do you understand grounded generation and abstention?
I instruct the model to answer strictly from the provided chunks and to cite which chunk each claim comes from, then I verify the citations actually support the claims — an unsupported sentence is a faithfulness failure I can catch automatically. Crucially, I reward abstention: if the context doesn't contain the answer, "I don't have that information" is the correct output. Penalizing refusals trains the system to hallucinate.
Weak answer: "Tell it not to hallucinate in the system prompt." No citations, no faithfulness check, no abstention.
8. "Your recall@5 is 0.95 but users still complain. What's going on?"
Really probing: can you reason past a single aggregate metric?
A good average can hide a bad slice. I'd break recall down by query type and document source — recall@5 might be 0.99 for how-to questions but 0.60 for exact-identifier lookups, or great for one product's docs and poor for a recently added one. I'd also check the generation side: retrieval can be fine while grounding, citation, or abstention fails. The aggregate says "mostly fine"; the slices say where it isn't.
Weak answer: "Recall is high, so retrieval is solved." Trusts one number and stops looking.
9. "What breaks when this goes to production?"
Really probing: have you thought past the notebook?
The corpus changes — new docs, edits, deletions — so I need incremental re-indexing and a way to catch stale answers. Query distribution drifts away from my gold set, so I sample real production queries and fold them back into evaluation. Latency and cost become constraints, so I tune candidate-set size and reranker depth. And I monitor online signals — abstention rate, citation-support rate, thumbs-down — because offline recall doesn't catch everything.
Weak answer: "It should just work like it did locally." No re-indexing, no drift, no monitoring.
10. "Sell me on a piece of RAG work you've done."
Really probing: do you have a real, measured artifact?
Walk through your portfolio pipeline: the corpus, the failure you found (e.g. exact-identifier queries missing because you were dense-only), the fix (added BM25 + RRF), and the measured result (recall@5 on those queries went from 0.55 to 0.9 on a 500-query gold set). Concrete corpus, concrete failure, concrete fix, concrete numbers — that's the whole interview.
Weak answer: "I read about RAG and understand the concepts." No artifact, nothing measured, nothing to probe.