What a Healthy Eval Setup Looks Like
You don't need to build the system — but you should recognize one that's working. A healthy setup has two halves: a gate before you ship, and a watch after.
Offline evals run before release, on a fixed set of cases, answering "is this change good enough to ship?" Online evals run on live traffic, answering "is it actually working out there?" You need both: offline covers the cases you anticipated; online catches the ones you didn't. Underneath, every eval is the same four parts — a dataset (the cases), a task (running the system on them), a scorer (turning each output into a number), and a report (rolling it up, broken out by segment).
Two teams, same model
Team A ships when "it looks good to the engineer." Team B has an automated eval that blocks a release if quality drops below a baseline, plus a live quality dashboard. Both have shipped for six months. Which team's roadmap can you actually trust, and why?
Team B — and it's not close. Team A's quality is invisible: it could be drifting down right now and no one would know until customers complain. Their velocity is fake, because every change is an unmeasured risk. Team B can move fast safely: the gate means a bad change is caught before users see it, and the dashboard means regressions surface in hours, not quarters. As a leader, fund the gate and the dashboard early — they're not overhead, they're what makes the rest of the roadmap believable.
HelpStack: three weeks of silent decay
HelpStack had a pre-release check but no live monitoring. A routine update quietly broke part of their system, and quality dropped for a whole category of questions — nothing errored, the assistant just started answering worse. With no dashboard watching live quality, the regression ran for three weeks before a support manager spotted the pattern in escalations.
The post-mortem was blunt: their offline check hadn't covered the affected cases, so it passed clean the entire time. They added a live dashboard tracking quality by segment; the next such break was caught in hours. The lesson for a leader: the offline gate proves a change is good on the cases you thought of, and only online monitoring catches the ones you didn't — you need both, and funding the dashboard is not optional overhead.
This chapter: before launch, Meridian funds two things the CEO first calls "nice to have" — an automated check that blocks any Remi release if quality drops below a baseline, and a live dashboard showing answered / abstained / wrong-action rates. Both earn their keep inside the first month, when a well-meaning change would otherwise have shipped a regression.
Quiz · Chapter 2
- A healthy eval setup has:
- The four parts of any eval are:
- "We test by hand before each release" is:
- The reason to fund evals early is: