Evaluating Agents

Agents are unusually hard to measure. The same task can succeed through a good path or a lucky one, and fail through a wrong answer or a thirty-step slog. Evaluating agents means scoring both the result and the path, with a harness you build yourself.

Result and path

Outcome evaluation asks whether the agent reached the goal. For tasks with a checkable end state ("the order exists," "the right number came back," "the file contains X"), check it in code. Asking a model "did it succeed?" is softer and easier to fool.

Trajectory evaluation asks how it got there. Two agents can both succeed while one takes 3 steps and the other 18, or one makes five redundant calls. Scoring only outcomes hides this, and you find out from the bill.

What to measure

Report task success rate, steps per task (a stand-in for cost and time), cost per task, and a breakdown of why failures happen: wrong tool, bad arguments, gave up, wrong final answer. The breakdown tells you what to fix.

A fixed task set

Build a set of representative tasks with checkable outcomes and freeze it, so every change is a before-and-after on the same tasks. Twenty to fifty tasks is enough to answer the questions that matter: did this prompt change raise success, or just move it around? Did adding a review step (Chapter 11) help enough to justify its cost?

Run it more than once

The same agent on the same task can take a different path, and sometimes get a different result, on a second run. One run isn't a measurement; a rate over a set is. For important tasks that are flaky, run each a few times and report the pass rate. Calling an agent "working" after one lucky success is the most common mistake in agent evaluation.

Starter code

TASKS = [f"task{i}" for i in range(8)]

# Recorded outcomes: (succeeded, steps) per task, per config.
RESULTS = {
  "lean":     [(True,4),(True,3),(True,5),(False,6),(True,4),(True,3),(True,4),(False,3)],
  "thorough": [(True,14),(True,16),(True,15),(True,18),(True,13),(False,20),(True,15),(True,14)],
}

def evaluate(cfg):
    rows = RESULTS[cfg]
    success = sum(ok for ok,_ in rows)/len(rows)
    steps   = sum(s for _,s in rows)/len(rows)
    return success, steps

# TODO Step 3: print both, then argue the ship decision on success AND steps.

for cfg in ("lean","thorough"):
    s, st = evaluate(cfg)
    print(f"{cfg:9} success {s:.0%}  avg steps {st:.1f}")
# => lean      success 75%  avg steps 4.0
#    thorough  success 88%  avg steps 15.6