Evaluating Agents

Agents — systems that plan, call tools, and act over many steps — are the hardest thing to evaluate, because the output isn't a single text but a trajectory: a sequence of decisions, tool calls, and observations. This is the newest, fastest-growing frontier of the role.

Outcome vs. process

Outcome / task success — did it reach the goal? (Booked the meeting, resolved the ticket.) The cleanest signal, often a verifiable end-state. Process / trajectory — how did it get there? Two agents can both succeed, but one took 3 clean steps and the other took 30 with destructive detours. Trajectory eval asks: right tools, valid arguments, recovered from errors, avoided unnecessary or dangerous steps, stayed efficient?

What to measure — and why it's hard

Task success rate, tool-use correctness, efficiency (steps, tokens, latency, dollars — cost and latency are first-class quality metrics for agents), and robustness to bad inputs. The deep difficulty: errors compound.

A "95%-reliable" agent is a coin flip by 20 steps. That's why realistic agent benchmarks need resettable sandbox environments, not static datasets — and why building those harnesses is core to the job.

Lab code — runs free

RUNS = [   # each step is (tool, succeeded?)
    {"steps": [("search", True),  ("read", True),  ("reply", True)],  "resolved": True},
    {"steps": [("search", True),  ("delete", True),("reply", True)],  "resolved": True},
    {"steps": [("search", True),  ("read", False), ("reply", True)],  "resolved": False},
]
ALLOWED = {"search", "read", "reply"}

def outcome(r):       return float(r["resolved"])
def trajectory_ok(r): return all(tool in ALLOWED and ok for tool, ok in r["steps"])

success = sum(outcome(r)       for r in RUNS) / len(RUNS)
clean   = sum(trajectory_ok(r) for r in RUNS) / len(RUNS)
print(f"task success: {success:.0%}    clean trajectory: {clean:.0%}")

# errors compound across steps:
for n in (1, 5, 10, 20):
    print(f"  {n:>2} steps @ 0.95/step -> {0.95**n:.0%} end-to-end")

task success: 67%    clean trajectory: 33%
   1 steps @ 0.95/step -> 95% end-to-end
   5 steps @ 0.95/step -> 77% end-to-end
  10 steps @ 0.95/step -> 60% end-to-end
  20 steps @ 0.95/step -> 36% end-to-end