Interview Simulation
Ten questions you're likely to be asked, what each one is really testing, a strong answer built from this book and the kind of answer that loses the room. Practice saying them out loud.
Evaluation interviews rarely ask for definitions. They hand you a vague product and watch whether you can turn "is it good?" into a plan with datasets, scorers, slices and sample sizes, and a way to tell improvement from noise. Two habits help in almost every answer: walk through the four parts of an eval, and mention slices and sample size. Bring your capstone project too. Showing it beats describing it.
1. "Walk me through how you'd evaluate a new customer-support chatbot."
What they're testing: whether you can turn a fuzzy "good" into criteria and a plan.
I'd start by naming what good means here: resolving the issue correctly, staying faithful to the knowledge base, tone, and safety. Then I'd build a dataset from real tickets, sliced by topic, language, difficulty and customer tier. I'd use deterministic checks where I can, such as policy compliance and format, and a model judge for the subjective parts, checked against human labels on the hard cases. Offline, that's the release gate. Online, I'd run the judge on sampled traffic and watch signals like ratings and regenerations. I'd report overall and by slice, always with the sample size.
Avoid: "I'd read some responses and see if they look good," or one accuracy number with no slices.
2. "You changed a prompt and it seems better. How do you know?"
What they're testing: rigor.
I'd measure a baseline on a fixed eval set, run the change and compare. For "did it help?" I prefer a pairwise judge with swapped order, because comparisons are more reliable than absolute scores. I'd check the slices, since the average can rise while an important segment drops. I'd make sure the gap is bigger than the noise for my sample size, and then add the check to CI so it can't slip back later.
Avoid: "It looked better on a few examples."
3. "How do you trust an LLM-as-judge?"
What they're testing: whether you see the judge as a model with biases.
First, I check it against human labels, focusing on the hard cases where decisions get made, since a blended average is dominated by easy ones. Second, I correct for known biases: swap the order and keep only agreeing verdicts, tell the rubric not to reward length, and use a small label set instead of 1 to 10. Third, I check it again whenever the rubric or the underlying model changes.
Avoid: "It's a strong model, so its scores are accurate."
4. "How would you evaluate a RAG system?"
What they're testing: whether you separate retrieval from generation.
I'd evaluate retrieval and generation separately and then together. Retrieval is standard search evaluation: recall@k and nDCG against documents known to be relevant. For generation, given good context, I'd use a judge for faithfulness and answer relevance. For each wrong answer I'd decide which stage failed before proposing a fix, because if recall is low, no prompt change will help. I'd also track the faithfulness and helpfulness trade-off and pick a balance that fits the use case.
Avoid: scoring only the final answer with one metric like BLEU.
5. "How do you evaluate an agent?"
What they're testing: whether you look at the path as well as the result.
I'd measure outcome, meaning success on tasks with checkable end states, and trajectory: correct tool use, no destructive or wasteful steps, efficiency. Two agents can both succeed while one takes a dangerous shortcut. Cost and latency count as quality for agents. I'd also account for compounding error, since 95% per step is only about 36% over twenty steps, which is why agent tests need resettable sandboxes.
Avoid: only checking whether the agent finished.
6. "A provider updated their model and quality dropped. How would you have caught it?"
What they're testing: whether you understand production evaluation.
With online monitoring on top of offline tests: a judge on sampled traffic, implicit signals, drift detection such as PSI on the score distribution, and a regression suite on a schedule so a provider change trips an alert. Failures found in production become offline cases that protect the next release. I'd also keep guardrails, which act on each response, separate from evals, which measure trends.
Avoid: "We'd notice when users complain."
7. "Your offline eval says 95% but users are unhappy. What's going on?"
What they're testing: dataset representativeness and the gap between offline and online.
Usually the eval set doesn't match reality. It may be too easy or contaminated by training data, or it isn't sliced and a good average is hiding a broken segment. Or live traffic has simply moved away from what the set covers. I'd slice the eval, compare it with real inputs, check the online signals and turn the actual complaints into new eval cases.
Avoid: "The users are wrong," or "it's an edge case."
8. "How big should an eval set be?"
What they're testing: basic statistics.
Big enough that the score is stable from run to run. The 95% interval is about 1.96 × √(p(1−p)/n), so halving the margin takes four times the data. In practice that's a few hundred cases for each slice you care about, and more if you're trying to detect a small improvement. A score reported without its sample size can't be interpreted.
Avoid: "Twenty examples is plenty."
9. "How would you red-team a feature before launch?"
What they're testing: worst-case thinking.
I'd build an attack suite across categories: jailbreaks, prompt injection (especially if the system reads outside content), personal-data leaks and harmful content. I'd automate attack generation so it scales, track a resistance rate across releases and make it a launch requirement. A system with 99% quality is still a liability if the other 1% is dangerous.
Avoid: "I'd try a few bad prompts by hand."
10. "You don't have years of LLM experience. Why you?"
What they're testing: whether you can present your background as a strength.
Evaluation is careful measurement, and that's what my background is: precision and recall, nDCG, relevance judgments, golden sets, slicing, baselines and experiment design. That covers about half of AI evaluation. The newer parts are LLM judges, agent evaluation, adversarial safety and production LLM monitoring, and I've practiced each of them. Here's a harness I built.
This field is young, so experience counts for less than in most roles, and I can do the work. Here's a harness I built. It scores a RAG system for faithfulness and retrieval recall@k, corrects judge position bias by swapping the order, and gates on slices in CI. Can I walk you through it?
Avoid: apologizing for what you haven't done, or listing courses with nothing built to show.
End design answers with "and I'd report it by slice, with a sample size, gated in CI." Interviewers are listening for whether you treat quality as something you measure and defend. Have your capstone open on your laptop.