AI Engineer Dojo All courses
Evaluation

The AI Evaluation Engineer

Twelve chapters on measuring LLM systems: datasets, scorers, model judges, human review, RAG and agent evals, safety, and monitoring once the system is live. Each chapter closes with notes for new grads and for the people managing the work.

  1. Start Here

    Before You Start

    This book is written for engineers who will build evaluation systems. Most chapters also end with two short sections for readers who come at the…

    Free preview
  2. Ch 1

    Why LLMs Broke Testing

    For fifty years, testing software meant running the code and checking that the output matched what you expected. Put a language model in the loop…

    Free preview
  3. Ch 2

    The Four Parts of Every Eval

    Once you accept that quality is a measurement, you need to decide what to measure and when. This chapter gives you a small mental model that every…

    Free preview
  4. Ch 3

    Datasets & Ground Truth

    When an eval gives a misleading number, the dataset is usually the reason. Metrics get most of the attention, but the cases you choose decide…

    Free preview
  5. Ch 4

    Metrics & Scorers

    A scorer turns an output into a number or a label. Picking one means matching it to the task and being clear about what it can't see. No scorer…

    Free preview
  6. Ch 5

    LLM-as-Judge

    When there's no answer key and "good" is a matter of judgment, you ask a capable model to do the grading. It's the technique an evaluation engineer…

    Free preview
  7. Ch 6

    Human Evaluation & Annotation

    Automated scorers and model judges all rest on human judgment in the end. For subjective quality, people are still the ground truth, and the…

    Free preview
  8. Ch 7

    Evaluating RAG Systems

    Retrieval-augmented generation (RAG) means fetching documents first and then answering from them. It's the most common way LLMs are used in…

    Free preview
  9. Ch 8

    Evaluating Agents

    Agents plan, call tools and take several steps to finish a task. They're the hardest systems to evaluate, because the output is a whole trajectory…

    Free preview
  10. Ch 9

    Safety, Robustness & Red Teaming

    Quality evals ask whether the system is good. Safety evals ask whether someone can make it cause harm, and how it behaves under attack. Instead of…

    Free preview
  11. Ch 10

    Production Evaluation

    Offline evals tell you about the cases you thought of. Production shows you everything else. Online evaluation measures behavior on real traffic and…

    Free preview
  12. Ch 11

    Eval-Driven Development

    The earlier chapters were techniques. This one is about how an evaluation engineer works day to day, and how those techniques fit into a team's…

    Free preview
  13. Ch 12

    Tools, Interviews & Hiring

    You don't need to memorize tools. You need to know the categories well enough to map any company's stack onto ideas you already understand. Then you…

    Free preview
  14. Appendix

    Interview Simulation

    Ten questions you're likely to be asked, what each one is really testing, a strong answer built from this book and the kind of answer that loses the…

    Free preview

All courses · aiengineerdojo.com