Interview Simulation
Ten questions you will actually be asked, each with what the interviewer is really probing, a strong answer built from this book, and the weak answer that ends the interview. Practice saying these out loud.
AI-evaluation interviews rarely ask you to recite definitions. They hand you a vague product and watch whether you can turn "is it good?" into a measurable plan — datasets, scorers, slices, sample sizes, and a way to tell improvement from noise. The single best move in any answer below is to name the four parts of an eval (dataset → task → scorer → report) and always mention slices and sample size. And bring a portfolio: the little eval harness you built in Chapter 12 is a better answer than any sentence.
1. "Walk me through how you'd evaluate a new customer-support chatbot."
Really probing: can you convert a fuzzy "good" into measurable criteria and a plan?
First I'd name the quality dimensions — resolution/accuracy, faithfulness to the knowledge base, tone/brand, and safety — because "good" is several things. Then a representative, sliced dataset built from real tickets (by topic, language, difficulty, customer tier), not cherry-picked easy ones. Deterministic scorers where I can (did it follow policy, return valid format), an LLM-judge for the subjective dimensions, validated against human labels on the hard slice. Offline as the pre-ship gate, plus an online judge on sampled live traffic and implicit signals like thumbs and regenerations. I'd report overall and by slice, with a sample size — a bare number isn't an answer.
Avoid: "I'd read some responses and see if they look good," or quoting a single accuracy number with no slices or n.
2. "You tweaked a prompt and it seems better. How do you actually know?"
Really probing: rigor versus vibes.
I measure a baseline on a fixed eval set first, then run the change and compare — and for "did it help?" I prefer a pairwise judge, because comparison is more reliable than absolute scoring. I check slices, because the average can rise while a key segment drops, which is a regression. I confirm the gap is bigger than the noise given my sample size, then wire the check into CI so it can't silently regress later.
Avoid: "It looked better on a few examples."
3. "How do you trust an LLM-as-judge?"
Really probing: do you know a judge is a model with biases, not an oracle?
Three beats. One, validate it against human labels — and on the hard, decision-relevant slice, not a blended average that's dominated by easy cases. Two, mitigate the known biases: position bias with order-swap-and-agree, verbosity by telling the rubric not to reward length, and use a low-cardinality output instead of a fuzzy 1–10. Three, re-validate whenever the rubric or the underlying model changes — calibration doesn't transfer.
Avoid: "It's a strong model, so its scores are accurate."
4. "How would you evaluate a RAG system?"
Really probing: do you separate the two failure surfaces?
RAG has two surfaces and I evaluate them separately, then together. Retrieval is classic IR — recall@k and nDCG against a set with known relevant documents. Generation, given good context, I score with a judge for faithfulness and answer relevance. On any wrong answer I attribute it to retrieval versus generation before proposing a fix, because low recall means the facts never reached the model and no prompt change will help. I'd also track the faithfulness-versus-helpfulness trade-off and pick the operating point for the use case.
Avoid: scoring only the final answer with a single metric like BLEU.
5. "How do you evaluate an agent?"
Really probing: do you grade the trajectory, not just the end state?
Two layers: outcome — a verifiable end-state success rate — and trajectory — tool-use correctness, no destructive or wasteful steps, efficiency. Two agents can both "succeed" while one took a dangerous shortcut, so outcome alone isn't enough. Cost and latency are first-class quality metrics for agents. And I account for compounding error — 95% per step is a coin flip by twenty steps — which is why agents need resettable sandbox tasks, not static datasets.
Avoid: only checking whether the agent "finished."
6. "A provider silently updated their model and quality dropped. How would you have caught it?"
Really probing: do you understand production/online evaluation?
Online monitoring, not just offline tests. A sampled online judge plus implicit signals, drift detection like PSI on the score distribution, and a regression suite re-run on a schedule so a provider change trips a gate. The key idea is the flywheel — production surfaces failures, those become offline cases that guard the next release. And I'd separate guardrails, which enforce in real time, from evals, which measure over time.
Avoid: "We'd notice when users complain."
7. "Your offline eval says 95% but users are unhappy. What's going on?"
Really probing: dataset representativeness and the offline/online gap.
Usually the eval set isn't telling the truth: it's unrepresentative — too easy, or contaminated by training data — or it isn't sliced, so a great average hides a broken hard slice. Or the offline distribution simply doesn't match live traffic. I'd slice the eval, compare it to the real input distribution, check online signals, and mine the actual complaints into new eval cases so the suite reflects reality.
Avoid: "The users are wrong / it's an edge case."
8. "How big should an eval set be?"
Really probing: statistical literacy.
Big enough that the aggregate is stable run-to-run. The 95% interval is roughly 1.96·√(p(1−p)/n), so halving the margin needs four times the data. Practically that's a few hundred cases per slice you care about, and more if you're trying to detect a small improvement. The real point: a score reported without its sample size is meaningless.
Avoid: "Twenty examples is plenty."
9. "How would you red-team a feature before launch?"
Really probing: worst-case, adversarial thinking.
Build an adversarial suite across categories — jailbreaks, prompt injection (especially for RAG and agents, where untrusted content carries instructions), PII leakage, harmful content — and automate attack generation so it scales. I track a resistance rate and watch it release over release, and I gate launch on it. Safety is about the worst case, not the average — a 99%-quality system is still a liability if the 1% is dangerous.
Avoid: "Try a few bad prompts by hand once."
10. "You're coming from ML and search quality, not LLMs. Why you?"
Really probing: can you frame your background as an asset, not a gap?
Evaluation is rigorous measurement, and that's exactly what a search-quality and ML background is — precision/recall, nDCG, relevance judgments, golden sets, slicing, baselines, experiment discipline. That's half of AI evaluation already. The genuinely new surface is narrow — LLM-as-judge, agent/trajectory eval, adversarial safety, production LLM observability — and I've built each of those; here's a small eval harness I wrote. I'm not catching up to LLM teams; I bring the measurement rigor most of them are missing.
Avoid: apologizing for not having "years of LLM experience."
End every design answer the same way: "…and I'd report it by slice, with a sample size, gated in CI." Interviewers are listening for whether quality is something you measure and defend, not something you eyeball. Walk in with the Chapter 12 harness on your laptop — a candidate who built one is in a different tier than one who can only describe one.