Evaluating RAG / "Chat With Your Docs"
If you're building "chat with your documents," "ask our knowledge base," or any assistant grounded in your data, you're building RAG — and it fails in two distinct ways you must learn to tell apart.
RAG does two jobs: retrieval (find the relevant documents) and generation (write an answer from them). When an answer is wrong, the first question is always: did it fail to find the right documents, or did it have them and still answer badly? These have opposite fixes. If retrieval is the problem, no amount of prompt-tuning helps — the facts never reached the model. If generation is the problem, the model is hallucinating (inventing things not in the documents), which is the failure that gets companies in trouble. The metric for that is faithfulness: is every claim actually supported by the retrieved source?
Ask for three separate numbers: context relevance (did it retrieve useful material?), groundedness / faithfulness (is each answer claim supported by that material?), and answer relevance (did it answer the customer’s question?). TruLens calls this the RAG Triad. Then ask for retrieval precision and recall separately; those are RAGAS-style retrieval measures, not part of the Triad. A single "RAG accuracy" number hides the decision you need to make.
It confidently made something up
Your doc-assistant gave a customer a confident, wrong answer that wasn't in any of your documents. The team says "we'll improve the prompt." What do you ask before you accept that fix?
Ask: "Did retrieval even surface the right document?" Two very different bugs hide behind one symptom. If the right doc wasn't retrieved, the prompt is irrelevant — fix retrieval (how documents are chunked, indexed, ranked). If the right doc was retrieved and the model still invented an answer, that's a faithfulness/hallucination problem — fix grounding and add a faithfulness check. Also ask whether they're now measuring hallucination, so they'll know if it recurs. "We'll tweak the prompt" without that diagnosis is guessing.
Beacon: finding out where it actually broke
Beacon's "chat with your docs" assistant gave wrong answers, and the team was about to spend a sprint rewriting prompts. First they split the question in two: did the system even find the right passage, and — given the right passage — did it answer faithfully? The numbers were decisive: 40% of wrong answers were retrieval failures (the answer was never pulled up), while the generation step, handed the right passage, was faithful 94% of the time.
A single "accuracy" number would have hidden this and sent them polishing prompts that couldn't help. Because they measured the two stages separately, they fixed the real bottleneck — retrieval — instead. The leadership takeaway for any document-answering product: insist your team can tell you which stage failed, because "the AI is wrong" usually means a specific, fixable step upstream, not a bad model.
This chapter: Remi answers from Meridian's billing-policy documents, so it's a "chat with your docs" system underneath. On edge-case fee questions it cites the wrong fee schedule 18% of the time — traced to an outdated document still in the corpus, a retrieval problem, not Remi's wording. The fix is corpus hygiene, not a prompt tweak. (Ch 8: Remi begins taking refund actions.)
Quiz · Chapter 7
- RAG ("chat with your docs") can fail in two ways:
- "Faithfulness" measures whether:
- A wrong answer "so let's tweak the prompt" is premature until you know:
- The RAG failure that most often causes incidents is: