AI Search / RAG Engineer · Chapter 10

Production RAG

A RAG system that scored well offline can rot in production without a single code change — because the corpus changes, the query distribution drifts, and the index goes stale. Production RAG is the discipline of catching that decay from live signals before your users do.

Offline evals (Ch. 8) prove your system was good on a fixed gold set at one moment. Production is a moving target: docs get added and edited, users ask things your gold set never covered, and latency budgets bite. Three forces degrade a live system, each with a defense.

The three decay forces

Force	What happens	Defense
Staleness	Docs change; the index still serves the old chunks, so answers cite outdated facts	Incremental re-indexing on doc change; track index age; freshness SLAs
Query drift	Users ask new kinds of questions your gold set never had; recall silently drops on the new slice	Sample live queries into the gold set continuously; monitor per-slice
Latency budget	Hybrid + rerank + multi-query stack up; p95 latency blows the SLA	Cache embeddings & frequent queries; cap rerank candidates; parallelize retrievers

Online evals: measuring quality without labels

You can't hand-label every production query, so you lean on online signals that proxy quality: the answer-rate (how often the system refuses), citation-coverage (fraction of answers with valid supporting chunks), user thumbs / follow-up rate (a re-ask often means a bad first answer), and click/dwell on cited sources. None is ground truth, but a sudden move in any of them is a reliable alarm. The key move: alert on deltas, not absolutes. Citation-coverage dropping from 0.92 to 0.71 overnight means something broke — probably a re-index that mangled chunk ids — even without a single label.

The feedback loop that compounds

Every production query is a free future test case. Pipe low-confidence answers (refusals, thumbs-down, no-citation) into a review queue; the ones humans confirm as failures become new gold-set entries with the correct chunk labeled. Your eval set grows toward your real traffic over time, and each regression you catch makes the harness better at catching the next. This loop is what separates a system that decays from one that improves in production.

Try it · ~25 min

Detect a production regression from online signals — no labels

You'll take a week of daily online metrics and write the monitor that fires when quality regresses, without any ground-truth labels. You'll catch the day a bad re-index tanked citation-coverage — the way real on-call RAG engineers catch it.

Setup: none — pure Python.

Step 1. Compute a rolling baseline (mean of prior days) for each metric.

Step 2. Flag any day where a metric drops more than a threshold below baseline.

Step 3. Print the alert with the offending metric and day.

Your goal: the monitor fires on the regression day and names the metric — a labels-free alarm.

Starter code

# Daily online signals. citation_coverage = frac of answers with a valid supporting chunk.
# answer_rate = frac of queries answered (not refused). Day 4 had a bad re-index.
DAYS = [
  {"day":1, "citation_coverage":0.91, "answer_rate":0.78},
  {"day":2, "citation_coverage":0.92, "answer_rate":0.77},
  {"day":3, "citation_coverage":0.90, "answer_rate":0.79},
  {"day":4, "citation_coverage":0.71, "answer_rate":0.86},   # coverage crashed, answer_rate ROSE (answering w/o support!)
  {"day":5, "citation_coverage":0.70, "answer_rate":0.85},
]
DROP = 0.10   # alert if a metric falls >10% (absolute) below rolling baseline

# TODO: for each day from the 2nd on, baseline = mean of that metric over prior days;
#       if today < baseline - DROP, emit an alert naming the metric and day.

Worked solution

metrics = ["citation_coverage", "answer_rate"]
for i in range(1, len(DAYS)):
    for m in metrics:
        prior = [d[m] for d in DAYS[:i]]
        baseline = sum(prior)/len(prior)
        today = DAYS[i][m]
        if today < baseline - DROP:
            print(f"ALERT day {DAYS[i]['day']}: {m} {today:.2f} vs baseline {baseline:.2f}")

What you should see: ALERT day 4: citation_coverage 0.71 vs baseline 0.91. No labels were used — the monitor caught the regression purely from a delta against the rolling baseline. Now notice the trap: on day 4 answer_rate went up (0.79 → 0.86). A naive dashboard watching "are we answering questions?" would show green while quality collapsed. The system was answering more while supporting its answers less — it started confidently responding without grounding. Watching answer-rate alone would have missed it entirely; the pair of signals is what told the truth.

The production lesson: a single metric lies. Citation-coverage down + answer-rate up is a specific, recognizable failure signature (broken grounding after a bad index push). Alerting on the relationship between signals, on deltas rather than absolutes, is how you catch decay a static threshold would sleep through.

Going further: add a per-slice breakdown (identifier queries vs. how-to queries) and watch a regression that hits only the identifier slice — the fingerprint of a lexical-index build that silently failed while dense kept working. Slice-level monitoring localizes the outage to a component.

Case study

NewsHub: the index that froze in time

NewsHub's assistant answered beautifully in testing and failed the moment it mattered most: on breaking stories. The cause was mundane — a static index rebuilt nightly. For content less than 24 hours old, recall was about 0.20, because the answer-bearing article simply wasn't indexed yet. Their offline gold set, full of older articles, never caught it; the failure lived entirely in the gap between test data and live traffic.

The production fixes were operational, not algorithmic: incremental indexing on publish (fresh content searchable in minutes, lifting <24h recall to ~0.90), a freshness monitor, and — crucially — sampling real production queries back into the eval set, since live query distribution had drifted away from the curated gold set. Offline evals answer "is this change good enough to ship?"; only online signals answer "is it actually working out there?"

Running case · Acme Vault

Acme re-indexes changed docs on every release and folds 50 sampled production queries into its gold set weekly. That online loop catches a chunking regression from a docs-site migration that every offline eval had passed clean.

← Back Continue →