Chapter 10

Production RAG

A RAG system that tested well can decay in production without a single code change, because documents change, questions change and the index falls behind. Production RAG is about catching that from live signals before users do.

Case study

NewsHub's index froze in time

NewsHub's assistant tested well and failed on breaking news, the moment it mattered most. The index was rebuilt once a night, so for articles less than a day old, recall was about 0.20: the answer simply wasn't indexed yet. The offline gold set was full of older articles and never showed the problem.

The fixes were operational. They indexed articles as they were published, added a check on index freshness, and started sampling real production questions into the evaluation set, since the questions people asked had drifted away from the curated ones.

Three ways a live system decays

Cause	What happens	Defense
Staleness	Documents change and the index still serves old chunks	Re-index on change, track index age, set freshness targets
Query drift	Users ask new kinds of questions and recall drops on them	Sample live queries into the gold set; monitor by slice
Latency	Hybrid, reranking and multi-query add up and blow the time budget	Cache embeddings and common queries, cap rerank candidates, run retrievers in parallel

Monitoring without labels

You can't label every production query, so you watch signals that track quality: the answer rate (how often the system refuses), citation coverage (the share of answers with valid supporting chunks), thumbs and follow-up questions (a re-ask often means the first answer was bad), and clicks on cited sources. None of these is ground truth, but a sudden change in any of them is a reliable alarm. Alert on changes against a recent baseline rather than fixed thresholds. If citation coverage drops from 0.92 to 0.71 overnight, something broke, probably a re-index that mangled chunk IDs.

The loop

Send low-confidence answers (refusals, thumbs-down, answers without citations) to a review queue. The ones people confirm as failures become new gold-set questions with the correct chunk labeled. Over time the evaluation set looks more like real traffic, and each regression you catch makes the next one easier to catch.

Try it · ~25 minFree · no API key

Catch a regression without labels

You have a week of daily metrics. Write a monitor that compares each day against the average of the days before it and flags any metric that drops more than a threshold. It should fire on the day a bad re-index hurt citation coverage and name the metric.

Starter code

# Daily online signals. citation_coverage = frac of answers with a valid supporting chunk.
# answer_rate = frac of queries answered (not refused). Day 4 had a bad re-index.
DAYS = [
  {"day":1, "citation_coverage":0.91, "answer_rate":0.78},
  {"day":2, "citation_coverage":0.92, "answer_rate":0.77},
  {"day":3, "citation_coverage":0.90, "answer_rate":0.79},
  {"day":4, "citation_coverage":0.71, "answer_rate":0.86},   # coverage crashed, answer_rate ROSE (answering w/o support!)
  {"day":5, "citation_coverage":0.70, "answer_rate":0.85},
]
DROP = 0.10   # alert if a metric falls >10% (absolute) below rolling baseline

# TODO: for each day from the 2nd on, baseline = mean of that metric over prior days;
#       if today < baseline - DROP, emit an alert naming the metric and day.

Worked solution

metrics = ["citation_coverage", "answer_rate"]
for i in range(1, len(DAYS)):
    for m in metrics:
        prior = [d[m] for d in DAYS[:i]]
        baseline = sum(prior)/len(prior)
        today = DAYS[i][m]
        if today < baseline - DROP:
            print(f"ALERT day {DAYS[i]['day']}: {m} {today:.2f} vs baseline {baseline:.2f}")

You should get an alert on day 4: citation coverage at 0.71 against a baseline of 0.91. No labels were needed. Look at answer rate on the same day: it went up, from 0.79 to 0.86. A dashboard watching only "are we answering questions?" would have looked healthy while the system was answering more and backing up its answers less. Falling citation coverage with rising answer rate is a recognizable pattern, typical of grounding breaking after a bad index push.

To go further, split the metrics by query type and simulate a regression that hits only identifier queries. That pattern usually means the keyword index failed to build while dense search kept working.

Running case · Acme Vault

Acme re-indexes changed docs on every release and adds 50 sampled production questions to the gold set each week. That loop catches a chunking regression from a docs-site migration that every offline check had passed.

If you're starting out

Class projects end at launch, so production questions are where new grads have the most to learn. Be ready to name the three decay causes and a label-free signal for each. That alone puts you ahead of most applicants.

The idea of alerting on changes instead of fixed thresholds applies everywhere in monitoring. It's worth being able to explain why with a simple example.

If you run the team

If complaints creep up months after launch and nobody changed the code, the usual causes are freshness and drift. Ask two questions: when a document changes, how long until the assistant knows? And when did real production questions last go into the test set? Neither problem is a bug; both come from treating launch as the finish line.

Cost belongs on the same dashboard. If usage doubles and the bill goes up six times, the usual cause is sending too much text with each question. Ask how much text goes to the model per question and whether it grows with the document set.

Back Continue