Measuring Whether It's Any Good
A single quality number is comforting and often a lie. The teams you can trust don't report one number — they report it broken down by the segments that matter, because a great average routinely hides a broken slice.
To measure a RAG system honestly you need a gold set: a collection of real questions, each paired with the passage that truly answers it, big enough to be stable and tagged by category. Run the system over it and you learn two things a demo never shows — how often it retrieves the right passage (retrieval quality) and how often the final answer is actually faithful to the documents (answer quality). Crucially, you slice the results: by product, by question type, by document age.
Why slicing is non-negotiable: an overall "90%" can be 98% on your well-documented flagship and 60% on the new product line — and it's the 60% generating the angry tickets. The average says "mostly fine"; the slices say where it isn't, which is the only thing that tells you where to invest.
The reassuring 90%
Your team proudly reports "90% answer accuracy" and asks to expand the rollout. What do you ask before you approve — and what are you afraid the average is hiding?
Ask: "90% on how many questions, and show me that number sliced by product, question type, and document age." You're afraid the average hides a weak segment — often the newest or most important one — running at 50–60% and generating most of the complaints. If they can't slice it, they don't actually know their quality; they know a comforting mean. Approve expansion when the weak slices are acceptable, not when the average is — because customers experience slices, not averages.
InsureBot: the 0.90 that hid a 0.62
InsureBot was ready to expand: their gold set of 300 questions showed retrieval recall@5 of 0.90. Before scaling, they did one more thing — sliced the number by question type. The average shattered: routine FAQ questions scored 0.98, but questions about newly added policy documents scored just 0.62. A quarter of real traffic lived in that weak slice, completely invisible in the blended figure.
Because the gold set was tagged by segment and large enough that even a subset of ~70 questions was stable, the weak slice was catchable rather than a mystery. They held the rollout, fixed retrieval for new documents, and expanded once the weak slice was acceptable. The takeaway their COO adopted as policy: never approve on an average — ask for the slices.
Brightline builds a 300-question gold set from real employee queries, tagged by document type. The first sliced report reads 0.95 / 0.91 / 0.64 across how-to, reference, and error-code questions — pointing the next sprint straight at error-code retrieval instead of the parts already working.
Quiz · Chapter 7
- A trustworthy quality report is:
- A gold set is:
- An overall "90%" is dangerous because:
- "We hit 95% on a public benchmark" is: