The AI Evaluation Engineer

1. Ch 1

   Why LLMs Broke Testing

   Fifty years of software testing rests on one move: run the code, assert the output equals the expected value. The moment a language model enters the…

   Free preview
2. Ch 2

   The Evaluation Stack: A Mental Model

   Once quality is a measurement, the next questions are what to measure and when. Mature teams use a layered stack — a probabilistic cousin of the…

   Free preview
3. Ch 3

   Datasets & Ground Truth

   The most common reason an eval lies to you is a bad dataset. Metrics get the attention; the dataset is where credibility is won or lost — an…

   Free preview
4. Ch 4

   Metrics & Scorers

   A scorer turns an output into a number or label. The art is matching the scorer to the task — and knowing each one's blind spots, out loud.

   Free preview
5. Ch 5

   LLM-as-Judge

   When there's no answer key and "good" is a matter of judgment, you make a capable model do the grading. It's the single most-used technique in the…

   Free preview
6. Ch 6

   Human Evaluation & Annotation

   Automated scorers and judges are scaffolding on top of human judgment. Humans remain the ground truth for subjective quality — and the eval engineer…

   Free preview
7. Ch 7

   Evaluating RAG Systems

   Retrieval-Augmented Generation — fetch documents, then answer using them — is the most common production LLM architecture. Evaluating it is where…

   Free preview
8. Ch 8

   Evaluating Agents

   Agents — systems that plan, call tools, and act over many steps — are the hardest thing to evaluate, because the output isn't a single text but a…

   Free preview
9. Ch 9

   Safety, Robustness & Red Teaming

   Quality evals ask "is it good?" Safety evals ask "can it be made to do harm, and does it fail gracefully under attack?" These are adversarial —…

   Free preview
10. Ch 10

    Production Evaluation: Monitoring & Online Evals

    Offline evals tell you about the cases you thought of. Production tells you the truth. Online evaluation closes the loop — measuring real behavior…

    Free preview
11. Ch 11

    Eval-Driven Development: The Workflow

    Everything so far is technique. This is the practice — how an eval engineer actually works, and the philosophy that ties the tools together.

    Free preview
12. Ch 12

    Tools of the Trade & Interview Prep

    You don't need to memorize a tool to interview well — you need the categories and vocabulary, so you can map any company's stack onto concepts you…

    Free preview
13. Appendix

    Interview Simulation

    Ten questions you will actually be asked, each with what the interviewer is really probing, a strong answer built from this book, and the weak…

    Free preview

The AI Evaluation Engineer · AI Engineer Dojo · Other editions