Reliability & Failure Handling
Demos follow the happy path. In production, tools time out, return garbage or contradict each other, and the model loops or invents a tool that doesn't exist. Reliability work is what separates an agent that impresses in a meeting from one you can leave running.
Nimbus and the retry storm
Nimbus's agent depended on a third-party API that failed now and then. The agent assumed every call succeeded, so a failure either crashed the run or, worse, led it to call the same failing endpoint every turn until the step budget ran out. The logs showed the same call repeated a dozen times.
They rebuilt the loop to expect failure: return a clear error the model could act on, retry temporary failures a limited number of times with backoff, and notice when the agent repeated a failed call so it could change approach or escalate. Completion under real, flaky conditions rose from 58% to 84%.
Tool failures are normal
Any tool that uses the network will fail sometimes. Don't let the exception crash the loop, and don't feed a raw stack trace to the model. Catch it and return a clean, structured error the model can reason about, such as {"error": "timeout", "retryable": true}, so it can retry, try another tool or report the problem. Temporary errors get a few retries with backoff. Permanent ones, like a 404 or an authentication failure, shouldn't be retried at all; retrying a 404 three times is three guaranteed failures.
Breaking the retry loop
The classic failure is a tool error followed by the same call again, and again. Beyond the step cap and repeat detection from Chapter 3, the key is to make errors informative. "ERROR" tells the model nothing. "ERROR: city 'Prais' not found. Did you mean 'Paris'?" lets it fix its own mistake. In one test with a flaky tool, switching to informative errors raised recovery from 31% to 79%.
Every error a tool returns is text the model will act on. Make it clear about what went wrong, suggest what to try, and say whether it's worth retrying.
Invented tools and bad arguments
A model may call a tool you never defined, or pass arguments that break the schema. Check every call against its schema before running it. If the tool doesn't exist or the arguments are wrong, return a helpful error ("There's no tool called send_sms; the available tools are …") instead of crashing.
Doing something twice
Because agents retry, a tool with side effects can run twice: charging a card, sending an email or creating a duplicate order. Make those tools idempotent, for example with an idempotency key so a repeated "create order #4471" does nothing the second time. This kind of bug never shows up in a demo and can be very costly in production. Read-only tools can be retried freely.
Measure what informative errors buy
Run an agent against a flaky tool twice: once with opaque errors, then with informative errors that name the problem, suggest a fix and say whether to retry. Count how many tasks recover each time. The stand-in model fixes its mistake only when the error explains it.
Starter code
TASKS = ["weather in Prais", "weather in Toyko", "weather in Berln",
"weather in Pariss", "weather in London"] # 4 typos, 1 clean
CITIES = {"paris","tokyo","berlin","london"}
def geocode(city, informative):
c = city.strip().lower()
if c in CITIES:
return {"ok": True, "temp": 14}
if informative:
near = min(CITIES, key=lambda x: _dist(c, x))
return {"ok": False, "error": f"'{city}' not found. Did you mean '{near}'?",
"retryable": True}
return {"ok": False, "error": "ERROR", "retryable": True}
def _dist(a,b): # crude closeness so the mock can suggest a fix
return sum(ch not in b for ch in a) + abs(len(a)-len(b))
def model_retry(task, informative):
# Self-corrects only if the error names the intended city.
first = geocode(task.split()[-1], informative)
if first["ok"]: return True
if informative and "mean" in first["error"]:
fixed = first["error"].split("mean '")[1].strip("'?.")
return geocode(fixed, informative)["ok"]
return False # opaque error: model just retries the same typo, fails
# TODO: recovery = mean(model_retry(t, informative) for t in TASKS), both ways.def rate(informative):
got = sum(model_retry(t, informative) for t in TASKS)
return got/len(TASKS)
print(f"opaque {rate(False):.0%}, informative {rate(True):.0%}")
# => opaque 20%, informative 100%With opaque errors, only the task that was already clean succeeds, 20%. With informative errors, the agent corrects each typo from the suggestion, close to 100% here. Real systems won't be that clean, but the direction matches the 31% to 79% result: same model, same tool, only the error text changed. To go further, add an authentication error marked as not retryable and confirm your loop doesn't retry it.
Ridgeline's payment API sometimes times out. The agent now retries a refund a few times with backoff and, if it still fails, escalates to a person with the details attached. Before, it either retried forever or reported a refund that never happened.
Retries, backoff and idempotency come from ordinary backend engineering, and knowing them well makes you stand out among candidates who have only built agent demos. If they're new to you, read about idempotency keys in a payments API's documentation. It's the clearest real-world explanation.
Ask the team what happens when a tool the agent depends on goes down. Good answers mention limited retries, clear errors and escalation to a person. Then ask which actions could accidentally happen twice, such as charges, emails or orders, and how they're protected. It's one of the most common and expensive production bugs with agents.
Quiz · Chapter 7
- A network tool throws an exception mid-loop. What's the right handling?
- Why did informative errors raise recovery from 31% to 79%?
- The agent calls a tool you never defined. What should the loop do?
- Why is "retry three times" wrong for a 404 or an authentication error?
- Why do tools that charge cards or send email need idempotency?