The Dataset Is the Asset
Teams obsess over which model and which prompt. The thing that actually determines whether your evaluation tells the truth is the dataset — and it's the part most often done badly.
An eval is only as honest as the cases it runs on. If those cases are easy or unrepresentative, you'll get a flattering number that has nothing to do with real customers. The fixes are unglamorous but decisive: make the set representative of real traffic, large enough to be stable, sliced by segment so you see where quality lives, and hard where it matters. Two more traps: a curated "golden" set rots as your product changes (someone must own and refresh it), and public benchmark numbers are often contaminated (the model trained on the test) — impressive and meaningless for your use case.
One team's eval of easy queries reported 92%. On real, representative traffic the same system scored 64%. Nothing changed but the dataset. If your team can't tell you how representative their eval set is, you don't actually know your quality.
The benchmark brag
A vendor (or your own team) proudly reports "95% on a standard industry benchmark." Why might that number be worthless for you — and what would you ask to see instead?
Two reasons it may be worthless: the benchmark may not resemble your traffic (your documents, your users, your edge cases), and public benchmarks are frequently contaminated — the model saw the answers in training, so the score is inflated. Ask to see performance on a private set built from your own real data, sliced by the segments you care about, with a sample size. "95% on a public benchmark" should never be a buying or launch decision; "82% on 500 of our own enterprise tickets" is.
Cortex: the 92% that was really 64%
Cortex reported 92% accuracy and was ready to scale. The number came from an eval set the team wrote themselves — clean, well-phrased questions, nothing like the messy, terse things real customers ask. When they rebuilt the set by sampling real traffic, the same system scored 64%. Nothing changed but the dataset, and the flattering 92% had nearly justified a big, wrong bet.
They fixed it by building the eval set from real usage, tagging it by segment, sizing each important slice to stay stable, and giving it a named owner to refresh. For a leader, the takeaway is a single diagnostic question: how representative is the eval set? If your team can't answer it, your quality number describes their test set, not your product — and the gap can be 28 points wide.
This chapter: Meridian builds Remi's eval set from real support tickets, tagged by intent. The refund slice is only ~12% of traffic but the highest-stakes, so they deliberately over-sample it — a blended average would bury the exact segment where a mistake costs money. A support lead owns the set and refreshes it monthly. (Ch 6: they pay for human labels on that slice.)
Quiz · Chapter 3
- The biggest determinant of whether an eval tells the truth is:
- A high public-benchmark score may be worthless for you because:
- A golden eval set should be:
- The best evidence of real quality is: