When It Meets the Real World
An agent that passes your tests can still stumble in production, because the real world is messier than any test suite and things drift. Launch is where operating an agent begins, not where it ends.
Three realities bite after go-live. Messy inputs: real requests are stranger than your test tasks, so completion drops and new failure modes appear. Flaky dependencies: the tools the agent relies on rate-limit and fail under load, so reliability handling that seemed optional becomes load-bearing. Drift: what users actually ask shifts over time, away from the tasks your suite was built on, so your measurements slowly stop reflecting reality.
The habits that keep an agent healthy in production are operational: full traces so every incident is explainable, monitoring of completion / steps / escalation rates so regressions surface fast, and regularly folding real production tasks back into the eval suite so it tracks what's actually happening. Offline tests answer "is this good enough to ship?"; only live signals and traces answer "is it still working, and what did it just do?"
Great in staging, shaky in production
The agent scored 82% completion in testing but is closer to 64% on real traffic, and no one can say why any given failure happened. What are the likely causes, and what capability is missing?
Likely causes: real inputs are messier than the test tasks (so completion drops), dependencies flake under load (so runs fail in new ways), and traffic has drifted from the suite. But the glaring gap is that "no one can say why" — that's missing traces. Require step-by-step tracing so every failure is explainable, add live monitoring of completion and escalations, harden the reliability handling for flaky tools, and sample real tasks back into the suite. You can't operate an acting agent you can't observe; visibility is the precondition for fixing the 64%.
Continuum: fine in staging, blind in production
Continuum's agent passed its suite at 82% and disappointed in production, where completion sat near 64%. Real tasks were messier, dependencies rate-limited under load, and — the killer — when something went wrong, the team had no trace of what the agent had done, so every incident was a guessing game that ate hours.
They invested in the unglamorous production layer: full traces of every step and decision, monitoring of completion / step-count / escalation, and sampling real production tasks back into the suite as traffic drifted. Mean-time-to-diagnose dropped from hours to minutes, and the live metrics caught regressions the offline suite missed. The lesson for leadership: an agent fails in ways a single output never reveals, so observability isn't optional — it's how you run one at all.
Tessera logs a full trace for every request and watches live completion and escalation rates, and folds 50 real requests into its suite each week. When completion dips after a tool update, the traces pinpoint the failing step and they fix it the same day — instead of discovering it in a pile of complaints.
Quiz · Chapter 10
- An agent can stumble in production because:
- The capability that makes incidents explainable is:
- To keep evaluation honest over time you should:
- "It passed testing, so it's fine" is: