Catching Silent Regressions in Production
The scariest AI failure makes no noise. Nothing crashes, no error logs, no alert — quality just quietly drops, and the first signal is angry customers or a churn spike weeks later. Offline tests won't catch it; only production monitoring will.
Your AI can get worse without anyone changing a line of code: the kinds of questions users ask shift, or a model provider updates the underlying model. Catching this requires watching live quality, not just uptime. Use explicit signals (thumbs-up/down, surveys) and implicit signals (users re-ask, edit heavily, copy a response, regenerate, or abandon), plus a sampled online judge and drift detection that flags when the quality distribution moves. The healthiest pattern is a flywheel: production surfaces a new failure → it becomes a permanent test case → the fix is proven → it can never silently come back. (A guardrail blocks a bad output in real time; an eval measures quality over time. You want both.)
Ask the team to put time to first token, end-to-end latency, error rate, cost per successful task, and throughput beside its quality metrics. A model routing plan is often the right answer: use a cheaper, faster model for routine work and escalate only ambiguous or high-risk cases to a stronger one. Do not accept a quality gain that destroys the response-time or unit-economics promise of the product without making that trade-off explicit.
"Did anything change?" "No."
Support tickets about your AI feature tripled this month. Engineering says "we didn't change anything" — and they're telling the truth. What likely happened, and what should already have been in place?
Two usual suspects: the input distribution shifted (users started asking new kinds of questions), or your provider silently updated the model under you. Both degrade quality with zero code changes — which is exactly why "we didn't change anything" is not reassurance. What should have been in place: live quality monitoring and drift alerts that would have caught the drop in hours, not a month of tickets. The fix going forward is the flywheel — turn these new failing tickets into permanent test cases so the next regression trips a gate instead of a customer.
Streamline: green offline, bleeding online
Streamline's assistant passed its pre-release eval at 89% and shipped. Weeks later, escalations were climbing though the offline number hadn't moved. The cause was drift: a product launch had changed what customers asked, pushing real traffic toward topics the frozen eval set barely covered. Offline it looked stable; online, quality on the new question mix had quietly fallen into the 60s.
They added online evaluation — monitoring live quality, sampling real conversations, and folding them back into the eval set on a schedule — and the regression became visible in days instead of a quarter. The leadership lesson: launch is the start, not the finish. Offline evals catch what you anticipated; only live monitoring catches what you didn't, and a static eval set slowly stops describing a moving product. Budget for the watch, not just the gate.
This chapter: Remi is live, and Meridian watches resolution, escalation, and wrong-action rates on real traffic. When they launch a new subscription tier, the question mix shifts and Remi's accuracy on the new billing topics slips — invisible to the frozen offline set, caught by sampling production tickets back in. They refresh the set and recover. (Ch 11: they make this a habit, not a heroic save.)
Quiz · Chapter 10
- A "silent regression" is dangerous because:
- Your AI can get worse with no code change because:
- The right safety net for this is:
- A guardrail differs from an eval in that it: