Evaluating Agents

You cannot improve what you can't measure, and agents are unusually hard to measure: the same task can succeed via a good path or a lucky one, fail via a bad answer or a 30-step slog. Evaluating agents means scoring both the outcome and the trajectory — and building the harness yourself.

Two complementary questions define agent evaluation:

Outcome eval: did the agent achieve the goal? For tasks with a checkable end state — "the order was created," "the correct number was returned," "the file contains X" — this is the ground truth. Prefer programmatic checks (assert the end state) over asking a model "did it succeed?", which is softer and gameable.

Trajectory eval: how did it get there? Two agents can both succeed while one used 3 steps and the other 18, one stayed on-budget and the other made 5 redundant calls. Outcome-only eval hides these — you ship an agent that works but costs 6× what it should, and you don't find out until the bill.

The metrics that matter

• Task success rate — % of tasks with the correct end state. The headline number.
• Steps per task — efficiency; a proxy for cost and latency (Ch. 1).
• Cost per task — total tokens × price; the number finance cares about.
• Failure taxonomy — why failures happen: wrong tool, bad args, gave up, wrong final answer. This is what tells you what to fix.

Build a fixed task set — your regression suite

The foundation is a set of representative tasks with checkable outcomes, frozen so every change is a before/after on the same tasks. Without it you're back to "it feels better." With even 20–50 tasks, you can answer the only questions that matter: did this prompt change raise success or just move it around? Did adding reflection (Ch. 11) improve quality enough to justify the extra steps it costs? The harness turns those from arguments into measurements.

Beware the non-determinism

Run the same agent on the same task twice and you may get different trajectories — and sometimes different outcomes. So a single run isn't a measurement; a rate over a set is. For flaky-but-important tasks, run each a few times and report success@k or the pass rate. Treating one lucky success as "it works" is the most common self-inflicted agent-eval error.

Starter code

TASKS = [f"task{i}" for i in range(8)]

# Recorded outcomes: (succeeded, steps) per task, per config.
RESULTS = {
  "lean":     [(True,4),(True,3),(True,5),(False,6),(True,4),(True,3),(True,4),(False,3)],
  "thorough": [(True,14),(True,16),(True,15),(True,18),(True,13),(False,20),(True,15),(True,14)],
}

def evaluate(cfg):
    rows = RESULTS[cfg]
    success = sum(ok for ok,_ in rows)/len(rows)
    steps   = sum(s for _,s in rows)/len(rows)
    return success, steps

# TODO Step 3: print both, then argue the ship decision on success AND steps.

for cfg in ("lean","thorough"):
    s, st = evaluate(cfg)
    print(f"{cfg:9} success {s:.0%}  avg steps {st:.1f}")
# => lean      success 75%  avg steps 4.0
#    thorough  success 88%  avg steps 15.6