The Evaluation Stack: A Mental Model
Once quality is a measurement, the next questions are what to measure and when. Mature teams use a layered stack — a probabilistic cousin of the classic test pyramid.
Two axes that organize everything
Offline vs. online. Offline evals run before deploy, on fixed datasets, in CI — fast, repeatable, answering "should I ship this change?" Online evals run on live traffic — answering "is it working in the wild?" Offline covers the cases you anticipated; online catches the ones you didn't.
Reference-based vs. reference-free. Reference-based compares output to a known-good answer. Reference-free judges an output on its own merits (faithfulness, format, helpfulness) because no reference exists.
The four parts of any eval
If you can name these four for a system, you understand its quality story: a dataset (the cases), a task/harness (how the system is invoked per case), a scorer (output → number/label), and an aggregation/report (scores → a decision, with slices and trends). Every framework you'll meet is an implementation of these four.
Push as much signal as possible down the stack — cheap, fast, deterministic where you can — and reserve expensive judges and humans for what truly needs them. An eval suite that takes an hour to run won't get run.
Build a minimal eval harness
You'll wire up all four parts — dataset, task, scorer, report — with a deterministic scorer and a stubbed task (no model call, no spend). Step 1: run it. Step 2: note the overall and the per-slice numbers. Step 3: the average looks fine — which slice is quietly failing, and would the headline number have told you?
Lab code — runs free
from statistics import mean
# 1. DATASET — tiny, tagged by slice
CASES = [
{"input": "2+2", "expected": "4", "slice": "easy"},
{"input": "12*12", "expected": "144", "slice": "easy"},
{"input": "17*23", "expected": "391", "slice": "hard"},
{"input": "31*29", "expected": "899", "slice": "hard"},
]
# 2. TASK — system under test (stub; swap in your real model call later)
def task(text):
fake = {"2+2": "4", "12*12": "144", "17*23": "390", "31*29": "899"}
return fake.get(text, "?") # note the wrong answer for 17*23
# 3. SCORER — deterministic: cheap, runs first, perfectly reliable here
def scorer(output, case):
return float(output.strip() == case["expected"])
# 4. REPORT — overall + by slice
def run_eval(cases, task, scorer):
rows = [{"slice": c["slice"], "score": scorer(task(c["input"]), c)} for c in cases]
by_slice = {s: mean(r["score"] for r in rows if r["slice"] == s)
for s in {r["slice"] for r in rows}}
return {"overall": mean(r["score"] for r in rows), "by_slice": by_slice, "n": len(rows)}
print(run_eval(CASES, task, scorer)){'overall': 0.75, 'by_slice': {'easy': 1.0, 'hard': 0.5}, 'n': 4}The headline 0.75 looks acceptable — but it's the average of a perfect easy slice and a coin-flip hard slice. The system is broken exactly where it's hard, and only the per-slice view shows it. That four-part skeleton (dataset → task → scorer → report-with-slices) is the spine of every eval you'll ever build; the rest of the book swaps in richer scorers and datasets.
Fathom: shipping blind for a year
Fathom shipped changes to their AI feature for a year with no evaluation stack — no fixed dataset, no scorer, no report. Each release was judged by an engineer trying a few prompts and deciding it "seemed fine." Quality drifted, and because nothing measured it, no one could say whether any given change helped or hurt. When churn spiked, they had no way to bisect which of fifty changes had caused it.
The fix was to install the four parts explicitly: a dataset of real cases, a task harness that runs the system on them, a scorer that turns each output into a number, and a report sliced by segment — plus an offline gate so quality couldn't silently drop. Within a quarter, "did this change help?" became a question with an answer. The lesson for their lead: the stack isn't overhead, it's the instrument that turns a year of guessing into a measurable roadmap.
This chapter: Meridian maps Remi onto the four parts. Dataset: real billing questions from support logs. Task: run Remi on each. Scorer: a mix — a deterministic check on refund amounts, a validated judge for tone. Report: accuracy sliced by intent. They wire an offline gate before each Remi change and a live dashboard after — so a regression can never ship or hide. (Ch 4: they choose the scorers.)
Quiz · Chapter 2
- Offline evals exist primarily to:
- "Reference-free" scoring means:
- The four parts of any well-formed eval are:
- The stack design heuristic is to:
- In the lab, the overall score was 0.75 but the system was badly broken because: