Datasets & Ground Truth
When an eval gives a misleading number, the dataset is usually the reason. Metrics get most of the attention, but the cases you choose decide whether the number has anything to do with your real users. If you've built relevance judgments for search, this chapter will feel familiar.
Cortex's 92% was really 64%
Cortex reported 92% accuracy for its support assistant and was planning a big expansion. The eval set had been written by the team: clear, well-phrased questions that looked nothing like what customers send. Real customers write terse, messy messages full of edge cases.
When the team rebuilt the set by sampling real traffic, the same system scored 64%. The model hadn't changed at all. They tagged the new set by segment, made each important slice large enough to be stable, and put one person in charge of refreshing it. The 28-point gap had come entirely from the old dataset.
What makes an eval set trustworthy
It should look like real traffic, with the same kinds of questions, formats and languages your users send. Clean, easy inputs give you a flattering number that doesn't predict anything. It needs to be large enough that the score doesn't jump around between runs, which is Chapter 1's lesson applied. Each case should be tagged by segment so you can see where quality is good and where it isn't. And it should include the hard, high-stakes cases on purpose, because the point of the set is to find problems.
Golden sets, contamination and synthetic data
A golden dataset is a curated set of cases with answers that people have checked. It's your yardstick, and it goes stale as your product and users change, so treat it like code: version it, give it an owner and review it on a schedule.
Watch out for contamination. If your eval cases appeared in the model's training data, which happens a lot with public benchmarks, the score will be inflated and won't tell you anything about your own use case. Private, recent or custom-built sets avoid the problem.
Synthetic data, meaning cases generated by a model, is useful for quickly covering lots of situations. It tends to reflect what the model finds natural, though, so use it to add breadth and keep human-curated cases as the anchor for what's true.
An eval made of easy queries told one team it was at 92%. The same system on representative traffic scored 64%. The dataset was the only thing that changed.
Use slices to find the hidden failure
You're given per-case scores already tagged by slice. Compute the overall score, then the score for each slice, and flag the weak ones. Pay attention to how different the two views look.
Lab code · runs free
from statistics import mean
# Per-case scores, already tagged by slice
SCORED = [
{"slice": "en", "score": 0.95}, {"slice": "en", "score": 0.92},
{"slice": "en", "score": 0.97}, {"slice": "en", "score": 0.90},
{"slice": "es", "score": 0.61}, {"slice": "es", "score": 0.55},
{"slice": "code", "score": 0.40}, {"slice": "code", "score": 0.48},
]
overall = mean(r["score"] for r in SCORED)
by_slice = {s: mean(r["score"] for r in SCORED if r["slice"] == s)
for s in {r["slice"] for r in SCORED}}
print(f"overall: {overall:.2f}")
for s, v in sorted(by_slice.items(), key=lambda kv: kv[1]):
flag = " <-- WEAK" if v < 0.70 else ""
print(f" {s:5} {v:.2f}{flag}")overall: 0.72
code 0.44 <-- WEAK
es 0.58 <-- WEAK
en 0.94You could ship on 0.72 overall. Split up, it's an excellent English system attached to a broken code path and a shaky Spanish one. Now imagine the eval set were 90% English. The blended number would climb toward 0.90 and you'd ship the broken parts without knowing. Getting a representative set and slicing it matter more than which metric you pick.
Meridian builds Remi's eval set from real support tickets and tags each one by intent: balance questions, refund requests and disputes. Refunds are only about 12% of traffic but they're where a mistake costs money, so the team deliberately adds extra refund cases until that slice has a few hundred. A support lead owns the set and refreshes it every month.
If you haven't worked with production data before, "representative" can sound abstract. A simple way to think about it: if you picked a random real user message and a random eval case, could you tell which was which? If the eval cases are noticeably cleaner, shorter or more polite, the set is too easy.
For a portfolio project, building the dataset well is one of the easiest ways to stand out. Pick a public dataset, explain how you sampled from it, tag the slices, and write two sentences on what it doesn't cover. Most student projects skip that step, and reviewers notice when you don't.
Teams spend a lot of time arguing about models and prompts. The eval dataset gets less attention and decides more. If your team can't tell you how closely its eval set matches real traffic, you don't really know your quality. Everything the team reports is about its test set.
Be careful with benchmark claims, from vendors or from your own people. A "95% on an industry benchmark" may come from questions nothing like yours, and public benchmarks are often contaminated. Ask for results on a private set built from your own data, split by the segments you care about, with the number of cases. "82% on 500 of our enterprise tickets" is something you can decide on.
Quiz · Chapter 3
- What most often makes an eval give misleading results?
- "Slicing" an eval set means:
- Data contamination refers to:
- What's the right use of synthetic eval data?
- Why did Meridian add extra refund cases to Remi's eval set?