Before You Start

This book is written for engineers who will build evaluation systems. Most chapters also end with two short sections for readers who come at the work from a different angle: people looking for their first AI job, and people who manage the engineers doing it.

What an evaluation engineer does

Plugging a language model into a product takes an afternoon now. Knowing whether the result is good enough to put in front of customers takes much longer, and most teams find out they can't answer that question until something goes wrong. The evaluation engineer is the person who can answer it with data. Is the assistant getting refunds right? Did last week's prompt change help, or did it just move the problems somewhere else? Would we notice if the model provider changed something tomorrow?

The job borrows from a few older ones. You'll use the statistics a data scientist uses, the pipelines and CI a software engineer builds, and the habit a product manager has of pinning down what "good" means for a real user. If you've worked on search quality or classic ML, you'll recognize a lot of it, and we'll point out where.

How each chapter is laid out

Every chapter opens with a short story about a team that measured something and didn't like what it found. Then comes the reading, with the numbers worked out so you can check them. After that there's a lab: a small Python file you run yourself, followed by the output you should see and what it means. A running story about a fintech called Meridian and its billing assistant, Remi, picks up in each chapter so you can watch one system grow from a demo into a monitored product.

The two closing sections are short. If you're starting out covers the background a new grad might be missing and how the topic comes up in entry-level interviews. If you run the team is for managers, founders and PMs: what a healthy version of this looks like, what should worry you, and the question to ask in your next review. Engineers will get something from both. Each chapter ends with a five-question test.

Running the labs

All the labs are in a public repo at github.com/kodabloomllc/ai-eval-labs. They use plain Python and run offline without an API key. A few of them can call a real model if you set USE_REAL_API = True and have a key, but the built-in mock is enough to learn from. Please run them. Reading the output of a lab you ran yourself sticks much better than reading ours.

Do you need a PhD?

No. You need to be comfortable reading and writing basic Python, and willing to meet a bit of statistics, which we explain from the beginning. Mostly you need the reflex of asking "how do we know?" when someone says a system works. That reflex is most of the job.