Chapter 11

Eval-Driven Development

The earlier chapters were techniques. This one is about how an evaluation engineer works day to day, and how those techniques fit into a team's routine.

Case study

Iterate's gate earns its keep

Iterate kept shipping changes that fixed one thing and broke another, because releases went out on the author's confidence. The team made every change to the AI system pass an offline eval in CI. If quality fell below the baseline, the merge was blocked, just like a failing unit test.

In the first month the gate blocked a prompt change that looked better on the three examples its author had tried. On the full eval set it dropped a key slice by nine points. Engineers also started moving faster, because they could make bold changes and trust the gate to catch a drop.

What eval-driven development means

It borrows from test-driven development. Before you tune a prompt or swap a model, you build the eval that defines what "better" means. Every change then becomes a small experiment: run the eval, compare with the baseline, keep what helps and drop what doesn't. Without it, prompt engineering turns into guesswork. You change something, it seems better, and three cases you didn't look at got worse.

The daily loop

Define the metric, build the dataset, measure a baseline, change one thing at a time while watching the slices, put the check in CI, and feed production failures back in. A few lessons come up again and again. Read real outputs regularly, because they show you problems no metric was designed to catch. Remember Goodhart's law: once a number becomes the target, people find ways to raise it without improving the product, so keep a held-out set nobody tunes against. Start small; twenty examples you've checked by hand are worth more than an elaborate automatic eval you don't trust. And track quality, safety, cost and latency together so you don't improve one by giving up another.

The role

You're part data scientist (statistics, datasets, error analysis), part software engineer (harnesses, pipelines, CI), part product thinker (turning "good" into criteria you can measure) and part experimenter (hypotheses, controls, evidence). The question you're there to answer is the one every AI team asks: is this working, and did our change help?

Try it · ~15 minFree · no API key

Gate a release in CI

Write the check that runs on every change: the overall score can't fall below the baseline, and no key slice can fall below its floor. The new version raises the average and should still fail. Work out why.

Lab code · runs free

BASELINE = 0.80                        # current production quality (in the repo)
SLICE_FLOOR = {"enterprise": 0.75}     # segments that must not regress

def mock_eval(version):
    # pretend we ran the full eval; v2 lifts the average but hurts enterprise
    return {
        "v1": {"overall": 0.80, "by_slice": {"enterprise": 0.78, "smb": 0.81}},
        "v2": {"overall": 0.83, "by_slice": {"enterprise": 0.71, "smb": 0.88}},
    }[version]

def check(version):
    r = mock_eval(version)
    assert r["overall"] >= BASELINE - 0.02, f"overall regressed: {r['overall']}"
    for s, floor in SLICE_FLOOR.items():
        assert r["by_slice"][s] >= floor, f"{s} regressed: {r['by_slice'][s]}"
    print(f"{version}: PASS")

for v in ("v1", "v2"):
    try:
        check(v)
    except AssertionError as e:
        print(f"{v}: FAIL -> {e}")

Worked solution

v1: PASS
v2: FAIL -> enterprise regressed: 0.71

v2's overall score of 0.83 beats v1's 0.80, so a gate that only checked the average would have let it through. It also dropped the enterprise slice from 0.78 to 0.71, below the 0.75 floor, while picking up easy gains on small business accounts. The slice floor catches a trade nobody meant to make. A baseline and slice floors checked into the repo and enforced on every change is eval-driven development in practice.

Running case · Meridian × Remi

Meridian puts Remi's eval in CI. Every prompt change, model swap or tool change has to pass the sliced eval, including the extra refund cases and the red-team suite, before it can merge. A well-meant change that made Remi chattier and worse at refunds is blocked automatically before any customer sees it.

If you're starting out

Goodhart's law comes up in interviews more often than you'd expect, usually disguised as "our metric kept going up but users weren't happier; what happened?" Explain that the team optimized the number itself, and suggest a held-out set and regular reading of real outputs.

Adding a CI gate to your portfolio project is easy and looks professional. A GitHub Actions workflow that runs your eval and fails when a slice drops below a floor takes an hour to set up, and it shows you think about evaluation as part of shipping software.

If you run the team

The best AI teams treat evaluation as the way they work, and you decide whether that happens. The shift is from "we changed it and it seems better" to "we measured it, and here's what moved." Make one question standard for every quality change: "Show me the slices." Suppose a change lifts overall quality from 0.80 to 0.83 but drops your enterprise customers from 0.78 to 0.71. That change should be rejected, because the average rose on easy cases while your most valuable users got worse.

Making quality visible helps more than any tool. One team put a shared dashboard of quality by segment on screen for five minutes at the start of every standup. Regressions became everyone's business on the day they appeared, and the pattern of fixing one thing while breaking another faded within a quarter.

Back Continue