Building an Eval Culture
The best AI teams don't treat evaluation as a phase — they treat it as the way they work. As the leader, you set whether quality is something the team measures and defends, or something it eyeballs and hopes.
The cultural shift is from "we tweaked it and it seems better" to "we measured it and here's the evidence." Concretely that means: every change is checked against a baseline before it ships; the check runs automatically so quality can't silently slide; and the team watches not just the headline number but the segments — because a change that lifts the average while sinking your biggest customer is a regression, not a win. Two warnings worth repeating to your team: look at the actual outputs (the most valuable habit; numbers hide what reading reveals), and beware gaming the metric — once a number becomes the target, people optimize the number instead of the product.
A prompt change lifts the overall score from 0.80 to 0.83 — looks like a win, ship it. But it quietly dropped the enterprise segment from 0.78 to 0.71, your highest-value customers. The average rose by padding easy cases. A team with a slice-aware gate catches this automatically; a team watching only the headline ships the regression.
The win that isn't
An engineer proposes shipping a change that raises overall quality 0.80 → 0.83. What's the one breakdown you ask to see before approving — and what would make you reject a "+3 point" win?
Ask for the per-segment breakdown. A higher average can hide a regression in a segment you care about — most dangerously your highest-value customers. If enterprise dropped while the average rose on easy cases, you reject the "win," because you'd be trading your most important users for a flattering number. Make "show me the slices, not just the average" the standing rule for every quality change. That one habit prevents a whole class of self-inflicted regressions.
The team that made eval a standup ritual
A product team kept shipping AI changes that fixed one thing and quietly broke another, because quality lived in scattered heads. Their manager made one change: a single shared dashboard — quality by segment — reviewed for five minutes at the start of every standup. Suddenly a regression was everyone's business the day it appeared, not a surprise in the next customer escalation.
Within a quarter, the "fixed A, broke B" pattern faded, because no change could silently move the shared number without someone noticing. Nothing about the technology changed; the manager had simply made quality visible and collective. The leadership lesson: eval culture isn't a tool you buy, it's a habit you protect — and five minutes a day on a shared number does more for reliability than any model upgrade.
This chapter: Meridian reviews Remi's quality dashboard for five minutes at the start of each weekly ops meeting, and mines real complaints into new test cases. Reliability stops being the eng lead's private worry and becomes a shared number the team owns — and it stops regressing, because now everyone would see it if it did. (Ch 12: build, buy, and who to hire to keep it running.)
Quiz · Chapter 11
- An eval culture replaces "it seems better" with:
- A change that raises the average but sinks a key segment is:
- The most valuable team habit in evaluation is:
- "Gaming the metric" means: