Landing Your First Eval Role

You've learned the skill and built a harness. Here's how to turn that into a job.

Build a portfolio — this is the whole game for new grads

Take the Chapter 12 framework and point it at a real, public dataset (Hugging Face has many). Build an eval that scores a real task, handles LLM-judge position bias, and reports by slice. Put it on GitHub with a short write-up: what you measured, what you found, what you'd fix. One real harness beats a page of buzzwords — and almost no other new grad will have one.

What to search for

Titles vary: AI Evaluation Engineer, LLM Evaluation, AI Quality Engineer, Applied AI (Evals), Model Quality, AI Test Engineer. Also look inside "AI Engineer" and "Applied ML" roles — eval is often a big part of the work even when it's not in the title.

What entry-level interviews reward

Rigor over jargon. Handed a vague "evaluate this chatbot," they want to hear: dimensions, a representative sliced dataset, deterministic scorers plus a validated judge, an offline gate plus online monitoring, and results reported with a sample size. The Interview Simulation appendix drills exactly this — practice it out loud.

Resume framing

Lead with what you built and measured, not courses taken: "Built an LLM-as-judge eval harness with position-bias mitigation; measured RAG faithfulness and retrieval recall@k on a 500-case sliced dataset." Numbers and verbs — that one line signals you can do the job.

A two-week plan

Week 1: read Ch 1–6, do every lab. Week 2: read Ch 7–12, do every lab, then build the portfolio harness on a real dataset. Rehearse the ten interview questions. Apply.