What a Healthy RAG Setup Looks Like
You don't need to build the pipeline — but you should recognize a working one. It has a shape, and a team that can't draw it for you probably can't debug it either.
Every "chat with your docs" system is the same assembly line: chunk the documents into passages, index them so they're searchable, retrieve the passages relevant to a question, optionally rerank them so the best land on top, and generate an answer from those passages with citations. When an answer is bad, a good team can tell you which stage failed. That single skill — localizing the failure — separates a team that improves from one that just swaps prompts and hopes.
Healthy setups also have two clocks. An offline check runs before each release on a fixed set of questions ("is this change good enough to ship?"), and online monitoring watches live traffic ("is it actually working out there?"). You need both: offline catches what you anticipated, online catches what you didn't.
Two teams, six months in
Team A ships when the assistant "looks good." Team B gates releases on a retrieval check and watches a live quality dashboard. Both have shipped for six months. Whose roadmap can you actually trust — and why?
Team B, and it isn't close. Team A's quality is invisible: the corpus changed, a dependency updated, and no one would know quality slipped until customers complained. Their velocity is fake because every change is an unmeasured risk. Team B moves fast safely — the gate catches a bad change before users see it, and the dashboard surfaces regressions in hours, not quarters. Fund the gate and the dashboard early; they aren't overhead, they're what makes the rest of the roadmap believable.
HelpStack: three weeks of silent decay
HelpStack had an offline check but no live monitoring. A routine update to how documents were processed quietly broke part of their indexing, so a slice of the help center stopped being searchable. Nothing errored; the assistant just started answering "I couldn't find that" — or worse, guessing — for those topics. Because no dashboard watched live outcomes, the regression ran for three weeks before a support manager noticed a pattern in escalations.
The post-mortem finding was blunt: the offline check hadn't included the affected documents, so it passed clean the whole time. They added an online dashboard tracking answered/abstained/wrong rates by topic; the next such break was caught in hours. Offline gates gave them confidence about the cases they'd thought of — only online monitoring caught the one they hadn't.
Before launch, Brightline funds two things the CEO initially called "nice to have": a check that blocks a release if retrieval quality drops below a baseline, and a live dashboard showing how many questions were answered with a citation, abstained, or flagged wrong. Both earn their keep within the first month.
Quiz · Chapter 2
- The stages of a RAG pipeline are roughly:
- The single most valuable diagnostic skill is:
- A healthy setup has:
- "It's just a vector database" is: