When the Documents Change
Your corpus isn't a snapshot; it's a living thing that gets edited, added to, and deleted from every day. A RAG system that was accurate at launch decays silently unless someone plans for change.
Two production realities bite leaders who thought launch was the finish line. First, freshness: if new or updated documents aren't re-indexed promptly, the assistant can't answer about anything recent — and this failure is invisible offline, where your test set is full of old content. Second, drift: what users actually ask shifts over time, away from the questions your gold set was built on, so your measurements slowly stop reflecting reality.
The habits that keep quality alive are operational, not clever: re-index documents when they change, monitor live outcomes, and regularly fold a sample of real production questions back into the gold set so your evaluation tracks what people are actually asking. Offline evals answer "is this change good enough to ship?"; only live signals answer "is it still working out there?"
"It was fine at launch"
Three months post-launch, complaints are creeping up though nobody changed the code. Your team is baffled. What are the two most likely culprits, and what do you ask them to check?
Almost certainly freshness or drift. Freshness: new or updated documents aren't being re-indexed, so the assistant is blind to anything recent — ask, "when a document changes, how long until the assistant knows?" Drift: users are asking new kinds of questions the system was never good at, invisible because the gold set is stale — ask, "when did we last add real production questions to our test set?" Neither is a code bug; both are the predictable cost of treating launch as the end instead of the start.
NewsHub: the index that froze in time
NewsHub's assistant answered beautifully in testing and failed exactly when it mattered — on breaking stories. The cause was mundane: the index rebuilt only nightly, so for content less than a day old, recall was about 0.20. The offline test set, full of older articles, never caught it; the failure lived entirely in the gap between test data and live traffic.
The fixes were operational: re-index on publish (fresh content searchable in minutes, lifting recent-content recall to ~0.90), a freshness monitor, and — importantly — sampling real production questions back into the evaluation set, since live queries had drifted from the original test set. Nothing about the model changed; the wins were entirely in treating the corpus and the eval set as living things.
Brightline wires re-indexing into its docs-publishing flow so an updated policy is searchable within minutes, and samples 50 real employee questions into the gold set each week. That weekly loop later catches a quality regression from a docs-platform migration that every offline check had passed clean.
Quiz · Chapter 8
- A RAG system accurate at launch can decay because of:
- Freshness failures are invisible offline because:
- To keep evaluation honest over time you should:
- "We indexed it once at launch" is: