AI Agent Engineer · Chapter 10

Production Agents

Everything so far makes an agent that works. Production makes it work at 3 a.m. under load, on a budget, when you're not watching. That means observability into every step, hard controls on cost and latency, and the boring infrastructure that turns a clever loop into a service.

Observability: you cannot debug what you can't see

An agent failure is a trajectory failure, so a single log line ("request failed") is useless. You need tracing: for each run, capture every step — the model's input and output, each tool call and its result, tokens and latency per step, and the stop reason. When a user reports "the agent gave a weird answer," you pull that trace and see exactly which step went wrong: a bad retrieval at step 3, a tool timeout at step 5, a hallucinated argument at step 7. Without traces, every agent bug is an unfalsifiable mystery; with them, it's a line item.

The minimum trace, per run

A run id; per step: the messages sent, the model's response (including tool calls), tool results, input/output tokens, latency, and the terminal stop reason (natural/stuck/capped). That's enough to reconstruct any failure and to compute your cost and step distributions. Log it structured, not as prose.

Cost control that actually holds

Two levers dominate the agent bill, both introduced earlier, now enforced:

Prompt caching. Your system prompt and tool definitions are large, identical every turn, and re-sent every turn (Ch. 1). Caching them means you pay full price once and a large discount on subsequent turns. For an agent with a 2,000-token system-prompt-plus-tools block looped over many turns, caching that static prefix can cut input cost on the repeated portion by a large fraction — often the single biggest line-item win available, for near-zero effort.

Hard budget caps. Beyond max_steps (Ch. 3), enforce a per-request token or dollar budget: if a run exceeds it, stop and return a partial result. This converts the tail-risk of a runaway request from "unbounded" to "bounded," which is the difference between a predictable bill and an incident.

Latency: stream, parallelize, and cache

Users tolerate a working agent far better when they can see it working. Stream the agent's progress — "searching…", "found 3 results, comparing…" — so the 12-second task feels like progress, not a hang. Parallelize independent tool calls (Ch. 2). And cache not just prompts but tool results where they're stable (the same lookup within a session shouldn't hit the network twice). Perceived latency is a product feature, and it's largely an engineering choice.

Concurrency and rate limits

In production you run many agents at once, and each can fan out to tools and models. You'll hit provider rate limits and downstream service limits. The infrastructure answer is unglamorous and essential: bounded concurrency, queues, backoff on 429s, and per-tenant quotas so one heavy user can't starve the rest. This is ordinary distributed-systems engineering — but agents make it acute because one user request can explode into dozens of model and tool calls.

Try it · ~25 min

Trace a run and enforce a token budget

You'll wrap the loop with per-step tracing and a hard token budget, then show the trace lets you pinpoint a failure and the budget caps a runaway. Both are production must-haves you can add in a few lines.

Setup: mock-only; a trajectory with one bad step and one runaway.

Step 1. Record a trace row per step (step#, tool, tokens, latency, note).

Step 2. Enforce a budget: stop when cumulative tokens exceed the cap; return partial.

Step 3. Print the trace and the stop reason; identify the failing step from the trace alone.

Your goal: a readable trace + "stopped: budget at step N" and the step you'd blame for the bad answer.

Starter code

STEPS = [   # (tool, tokens_this_step, latency_s, note)
  ("search",     900, 1.1, "ok"),
  ("read_doc",  1400, 1.4, "ok"),
  ("summarize", 1200, 1.2, "retrieved WRONG doc"),   # the real bug
  ("search",    1600, 1.5, "ok"),
  ("summarize", 1800, 1.6, "ok"),
  ("search",    2000, 1.7, "ok"),                    # would exceed budget
]
BUDGET = 6000     # total input tokens allowed per request

def run_traced():
    trace, total = [], 0
    for i,(tool,tok,lat,note) in enumerate(STEPS, 1):
        if total + tok > BUDGET:
            return trace, f"stopped: budget at step {i}"
        total += tok
        trace.append({"step":i,"tool":tool,"tokens":tok,"lat":lat,"note":note})
    return trace, "natural"

# TODO: print the trace rows, the stop reason, and the step whose note
# explains a bad final answer.

Worked solution

trace, reason = run_traced()
for r in trace: print(r)
print(reason)
bad = [r for r in trace if "WRONG" in r["note"]]
print("blame step:", bad[0]["step"] if bad else "none")
# => ... rows ...
#    stopped: budget at step 5      (cumulative tokens crossed 6000)
#    blame step: 3

What you should see: the run halts at the step where cumulative tokens cross 6000 — the budget converts a would-be runaway into a bounded, partial result — and the trace makes the real defect obvious: step 3 retrieved the wrong document, which is why the answer was bad, even though no step "errored." Without the trace you'd be guessing; with it, the culprit is a single row.

The engineering read. Tracing turns trajectory failures into line items, and budgets turn tail-risk into a bounded cost. Add prompt caching on the static system+tools prefix and streaming for perceived latency, and you've covered the four production levers — observability, cost, latency, and safety-of-spend — that separate a demo from a service.

Going further (optional): compute cost per run from the trace (tokens × price) and aggregate over many runs to get your cost distribution — the p50 and p99 that capacity planning and pricing actually depend on.

Case study

Continuum: fine in staging, blind in production

Continuum's agent passed its test suite at 82% and then disappointed in production, where completion sat closer to 64%. Real tickets were messier than the test tasks, dependencies rate-limited under load, and — the killer — when something went wrong, the team had no trace of what the agent had actually done, so every incident was a guessing game.

They invested in the unglamorous production layer: full traces of every step, tool call, and decision; online monitoring of completion rate, step distribution, and escalation rate; and sampling real production tickets back into the eval suite, since live traffic had drifted from the test set. With traces, mean-time-to-diagnose dropped from hours to minutes, and the online metrics caught regressions the offline suite missed. The lesson: an agent can fail in ways a single output never reveals, so observability isn't optional — it's how you operate one.

Running case · Ridgeline

Ridgeline logs a full trace for every ticket — each step, tool call, and decision — and watches live completion and escalation rates. When completion dips after a tool change, the traces show exactly which step broke, and they fix it the same day.

← Back Continue →