Hiring a RAG / AI Search Engineer

Five questions to ask candidates, what a strong versus weak answer sounds like, and a one-page scorecard. You don't need to build the system yourself — you need to evaluate the person who will.

1. "Our doc assistant gives a wrong answer about a document we definitely have. What do you check first?"

Strong: checks whether the answer-bearing passage was actually retrieved before touching the prompt; distinguishes retrieval failure (fix upstream) from generation failure (fix grounding). Weak: "I'd rewrite the prompt" or "use a bigger model" — tuning a stage that can't fix a missing passage.

2. "How would you measure whether this is any good?"

Strong: builds a gold set of real questions paired with the right passages, reports retrieval and faithfulness sliced by segment with a sample size, mines complaints into test cases. Weak: "I'd try some questions and see if it looks good," or a single number with no slices.

3. "Users search by exact error codes and it keeps missing them. Why, and what do you do?"

Strong: recognizes the exact-term blind spot of semantic search; adds keyword search and combines them (hybrid); measures the lift on identifier queries. Weak: "Upgrade the embedding model" — which won't fix a structural blind spot.

4. "How do you stop it from making things up, and how do you keep the bill sane?"

Strong: grounding, citations, and rewarded abstention for accuracy; retrieving only relevant passages (not dumping the corpus) for cost, and can reason about cost per question. Weak: "Tell it not to hallucinate," and no sense of what drives cost.

5. "Show me something you've built."

Strong: walks through a real pipeline — chunking, hybrid retrieval, reranking, grounded generation — with an eval harness and before/after numbers on a gold set. Weak: only names tools and courses, with nothing measured to show.