Tools, Interviews & Hiring
This chapter maps what you've learned onto the tools you'll name in interviews, walks through the questions you're likely to get, and closes with notes on landing a first RAG job and on hiring someone to do this work.
How Priya got the offer
Priya had never held the title "RAG engineer." She had one repo: a hybrid pipeline over 2,000 public documentation pages, with section-aware chunks, dense and BM25 retrieval fused with RRF, a cross-encoder reranker and cite-or-abstain generation, plus an evaluation harness reporting recall@k and faithfulness on a 500-question gold set. Her write-up opened with a result: dense-only retrieval scored 0.55 on identifier queries and hybrid scored 0.90, with the before-and-after shown by slice.
In the interview, asked how she'd evaluate a system, she opened her dashboard. Asked what she'd check first for a wrong answer, she walked through tracing it to a stage using her own logs.
Tools, by stage
| Stage | Tools to be able to name | How to choose |
|---|---|---|
| Chunking | LangChain and LlamaIndex splitters, Unstructured, custom structure-aware splitters | Prefer structure-aware; measure recall by size (Ch 3) |
| Embeddings | Provider embedding APIs; open models for on-premises | Fit the domain; same model for index and query (Ch 4) |
| Vector store | pgvector, Pinecone, Qdrant, Weaviate, Milvus | pgvector if you already run Postgres; managed if you need scale |
| Keyword search | Elasticsearch or OpenSearch BM25; rank-bm25 for small sets | Needed for hybrid (Ch 5) |
| Reranking | Hosted or open cross-encoder rerankers | Retrieve wide, rerank narrow (Ch 6) |
| Generation | Model provider SDK with structured output for citations | Ground explicitly; cite per claim (Ch 9) |
| Evaluation | RAGAS-style metrics, your own recall@k harness, a model judge for faithfulness | Separate stage metrics plus online monitoring (Ch 8, 10) |
"I treat RAG as a pipeline of stages I can measure. I trace every failure to a stage with separate evals, recall@k for retrieval and faithfulness for generation, and I only add complexity like hybrid search, reranking or agentic retrieval when the numbers show it helps."
Questions you'll hear
"Our RAG bot gives wrong answers. How would you debug it?" Start by checking, for the failing queries, whether the gold chunk was retrieved. If not, look at chunking, hybrid search and recall. If it was, look at grounding and citations. Don't change anything until you know which stage failed. The weak answer is "improve the prompt" or "try a bigger model."
"When would you use keyword search instead of vector search?" For exact identifiers, error codes, part numbers and rare jargon, where the token is the signal. In practice you run both and fuse them with RRF, because they miss different queries.
"Recall@50 is 0.94, but answers are still bad. What's wrong?" The answer is in the candidates but not near the top. Add a reranker and measure recall@3 and MRR before and after; expect recall@50 to stay the same. If the gold chunk is in the top 3 and answers are still wrong, it's a faithfulness problem.
"How do you know a change helped?" A fixed gold set of real queries, spread across query types, with labeled chunks. Every change gets a before-and-after on recall@k, MRR and faithfulness, with confidence intervals. In production, watch changes in citation coverage, answer rate and re-asks.
"The model hallucinates even with retrieval. What do you do?" Ground explicitly, require per-claim citations, check each claim against its chunk, treat refusals as correct, and put the top chunks at the edges of the context.
Acme's finished system has section-aware chunks, dense and BM25 retrieval fused with RRF, a cross-encoder reranker, cite-or-abstain generation, and a sliced gold set with live monitoring. You can now draw it on a whiteboard and defend each stage.
Build one real project. Priya's repo is a good template at any size. Pick a public documentation site, build hybrid retrieval with a reranker, write a gold set of 50 to 100 questions yourself, and report recall@k and faithfulness by query type. Put the numbers at the top of the README.
Search broadly. Titles vary: RAG Engineer, Search Engineer, AI Search, Applied AI Engineer, Retrieval Engineer, Knowledge Systems. Many "AI Engineer" roles are mostly RAG work even when the title doesn't say so.
Lead your resume with results. For example: "Built a hybrid BM25 and dense retriever with RRF and a cross-encoder reranker; raised identifier-query recall@5 from 0.55 to 0.90 on a 300-question gold set."
A two-week plan. Read Chapters 1 to 6 and do the labs in week one. Read 7 to 12, do the labs and build the project in week two. Practice the interview questions above out loud, then apply.
Build or buy. Off-the-shelf "chat with your docs" products get you running fast and are often right when the feature is a convenience. Building gives you control over quality, cost, security and the documents, which is worth it when answering from your documents is the product, when the data is sensitive, or when volume makes cost per question matter. A common middle path is to buy the search infrastructure and the model and build retrieval quality, evaluation and permissions yourself. Either way, measure the system's answers on your own gold set, including any vendor's.
Who to hire. You don't need a PhD. You need an engineer who thinks in retrieval quality, slices, faithfulness and cost. Ask candidates what they'd check first when the assistant gives a wrong answer about a document you have; strong ones check whether the passage was retrieved before touching the prompt. Ask how they'd measure quality, why exact error-code searches might fail, and how they'd keep the model from inventing things while keeping the bill reasonable. Then ask them to show something they've built.
A one-page scorecard. Rate each from 1 to 3. Do they trace failures to a stage before fixing anything? Do they measure with a gold set, slices and sample sizes? Do they know the exact-term blind spot and hybrid search? Do they treat refusals as correct and check citations? Can they reason about cost per question? Have they built something real?
Quiz · Chapter 12
- Asked how you'd debug wrong answers, what's the strongest first move?
- When would you reach for BM25 in particular?
- Recall@50 is high, answers are bad, and gold chunks sit near rank 30. What's the fix and its signature?
- Which habit best signals senior judgment in RAG work?
- What's a sensible default when deciding whether to build or buy?