AI Evaluation Engineer — New-Grad Edition

1. Start Here

   Is This Job For You?

   A five-minute orientation before the real chapters: what an AI Evaluation Engineer actually does, whether it fits you, and how to get everything out…

   Free preview
2. Ch 1

   Why LLMs Broke Testing

   Fifty years of software testing rests on one move: run the code, assert the output equals the expected value. The moment a language model enters the…

   Free preview
3. Ch 2

   The Evaluation Stack: A Mental Model

   Once quality is a measurement, the next questions are what to measure and when. Mature teams use a layered stack — a probabilistic cousin of the…

   Free preview
4. Ch 3

   Datasets & Ground Truth

   The most common reason an eval lies to you is a bad dataset. Metrics get all the attention, but the dataset is where credibility is quietly won or…

   Free preview
5. Ch 4

   Metrics & Scorers

   A scorer turns an output into a number or label. The art is matching the scorer to the task — and knowing each one's blind spots, out loud.

   Free preview
6. Ch 5

   LLM-as-Judge

   When there's no answer key and "good" is a matter of judgment, you make a capable model do the grading. It's the single most-used technique in the…

   Free preview
7. Ch 6

   Human Evaluation & Annotation

   Automated scorers and judges are scaffolding on top of human judgment. Humans remain the ground truth for subjective quality — and the eval engineer…

   Free preview
8. Ch 7

   Evaluating RAG Systems

   Retrieval-Augmented Generation — fetch documents, then answer using them — is the most common production LLM architecture, so it's the one you're…

   Free preview
9. Ch 8

   Evaluating Agents

   Agents — systems that plan, call tools, and act over many steps — are the hardest thing to evaluate, because the output isn't a single text but a…

   Free preview
10. Ch 9

    Safety, Robustness & Red Teaming

    Quality evals ask "is it good?" Safety evals ask "can it be made to do harm, and does it fail gracefully under attack?" These are adversarial —…

    Free preview
11. Ch 10

    Production Evaluation: Monitoring & Online Evals

    Offline evals tell you about the cases you thought of. Production tells you the truth. Online evaluation closes the loop — measuring real behavior…

    Free preview
12. Ch 11

    Eval-Driven Development: The Workflow

    Everything so far is technique. This is the practice — how an eval engineer actually works, and the philosophy that ties the tools together.

    Free preview
13. Ch 12

    Tools of the Trade & Interview Prep

    You don't need to memorize a tool to interview well — you need the categories and vocabulary, so you can map any company's stack onto concepts you…

    Free preview
14. Career

    Landing Your First Eval Role

    You've learned the skill and built a harness. Here's how to turn that into a job.

    Free preview
15. Appendix

    Interview Simulation

    Ten questions you will actually be asked, each with what the interviewer is really probing, a strong answer built from this book, and the weak…

    Free preview

AI Evaluation Engineer — New-Grad Edition · AI Engineer Dojo · Other editions