Hiring an AI Evaluation Engineer

Five questions to ask candidates, what a strong versus weak answer sounds like, and a one-page scorecard. You don't need to evaluate the AI yourself — you need to evaluate the person who will.

1. "How would you evaluate this product feature?" (give them a real one)

Strong: names quality dimensions, proposes a representative sliced dataset from real usage, picks scorers (deterministic where possible, a validated judge for subjective parts), wants both an offline gate and live monitoring, and reports with a sample size. Weak: "I'd run it on some examples and see if it looks good," or a single accuracy number.

2. "How do you know an AI judge is trustworthy?"

Strong: validate it against human labels on the hard cases; mitigate position and length bias; re-validate when the model or rubric changes. Weak: "It's a strong model, so it's accurate."

3. "Your offline eval says 95% but customers are unhappy. What's wrong?"

Strong: suspects an unrepresentative or unsliced dataset hiding a weak segment, or an offline/online gap; would slice, check live signals, and mine complaints into new test cases. Weak: "The customers are wrong" or "it's an edge case."

4. "How big should an eval set be?"

Strong: big enough that the number is stable; understands a small sample is noisy and a score without a sample size is meaningless; a few hundred per important segment. Weak: "Twenty examples is fine."

5. "Show me something you've built."

Strong: walks you through an actual eval harness — dataset, scorer, slices, maybe an LLM judge with bias handling. Weak: only describes tools and courses, with nothing built to show.

6. "How do you evaluate the quality and reliability of AI systems and their output? What tools or metrics do you use?"

Strong: gives you a system rather than a tool list: offline golden cases, deterministic checks, and a human-validated judge before release; online sampled traffic, explicit and implicit feedback, drift monitoring, and real-time guardrails; then operational latency, cost per successful task, and model-routing trade-offs. They name tools only after explaining what each measures. Weak: says "we use an LLM judge" or lists vendors without a dataset, validation plan, release gate, or production loop.