Production Agents

Everything so far produces an agent that works. Production means it keeps working at 3 a.m., under load and on a budget, while nobody's watching. That takes tracing of every step, firm limits on cost and time, and ordinary infrastructure that turns a clever loop into a service.

Tracing

An agent failure happens somewhere along a path, so a single "request failed" log line doesn't help. For every run, record each step: what was sent to the model, what came back (including tool calls), each tool result, tokens and time per step, and why the run ended. When a user reports a strange answer, you open the trace and see which step went wrong: a bad lookup at step 3, a timeout at step 5, an invented argument at step 7.

Controlling cost

Prompt caching. The system prompt and tool definitions are large, identical every turn, and resent every turn. Caching them means you pay full price once and much less on later turns. For an agent with a 2,000-token block of instructions and tools, it's often the biggest single saving available, for very little work.

Hard budgets. On top of the step limit from Chapter 3, set a token or dollar budget per request. If a run passes it, stop and return what you have. That turns the risk of a runaway request from unlimited into a known maximum.

Latency

People wait more patiently when they can see progress. Stream updates like "searching…" and "found 3 results, comparing…" so a 12-second task feels like work in progress. Run independent tool calls in parallel (Chapter 2), and cache tool results that don't change during a session.

Concurrency and rate limits

In production many agents run at once, and one request can turn into dozens of model and tool calls. You'll hit the provider's rate limits and those of your own services. The answers are standard: limit concurrency, use queues, back off on 429 responses, and set per-customer quotas so one heavy user can't slow down everyone else.

Starter code

STEPS = [   # (tool, tokens_this_step, latency_s, note)
  ("search",     900, 1.1, "ok"),
  ("read_doc",  1400, 1.4, "ok"),
  ("summarize", 1200, 1.2, "retrieved WRONG doc"),   # the real bug
  ("search",    1600, 1.5, "ok"),
  ("summarize", 1800, 1.6, "ok"),
  ("search",    2000, 1.7, "ok"),                    # would exceed budget
]
BUDGET = 6000     # total input tokens allowed per request

def run_traced():
    trace, total = [], 0
    for i,(tool,tok,lat,note) in enumerate(STEPS, 1):
        if total + tok > BUDGET:
            return trace, f"stopped: budget at step {i}"
        total += tok
        trace.append({"step":i,"tool":tool,"tokens":tok,"lat":lat,"note":note})
    return trace, "natural"

# TODO: print the trace rows, the stop reason, and the step whose note
# explains a bad final answer.

trace, reason = run_traced()
for r in trace: print(r)
print(reason)
bad = [r for r in trace if "WRONG" in r["note"]]
print("blame step:", bad[0]["step"] if bad else "none")
# => ... rows ...
#    stopped: budget at step 5      (cumulative tokens crossed 6000)
#    blame step: 3