Measuring Whether It Works
You can't judge an agent by watching it once, because an agent isn't one output — it's a multi-step process with many ways to go right or wrong. The teams you can trust measure completion over many tasks, and look at how it got there, not just the final result.
To measure an agent honestly you need a task suite: a set of realistic tasks each with a checkable success condition — did the ticket actually get resolved, did the record actually get written, did the answer actually match. Run the agent over all of them and you get the numbers that matter: completion rate (how often it finishes the job), average steps (how efficiently), and cost per task — with a sample size. A single demo run tells you none of this.
Crucially, good teams also evaluate the trajectory, not just the ending: did the agent take the right actions, avoid destructive mistakes, and stop when done? An agent can reach a correct-looking result via a reckless path — the process is part of the product. Ask for completion, steps, and cost over a real task suite, sliced by task type; anything less is judging a process by one lucky run.
"It works" — over how many?
Your team says the agent "works great" and wants to expand it. What do you ask, and what are you afraid one demo is hiding?
Ask: "over how many real tasks, what's the completion rate, how many steps on average, and what does it cost per finished task — sliced by task type?" You're afraid the demo hides a mediocre real completion rate (say 55%) and, worse, a slice of tasks where it takes harmful actions. Also ask whether they check the path, not just the ending, since an agent can reach a right-looking result recklessly. Approve expansion when the numbers over many tasks are acceptable and the failure modes are safe — not because one run looked great.
TaskForge: the demo said 100%, the suite said 55%
TaskForge was ready to ship on the strength of flawless demos. Then they built a real evaluation: 40 tasks with checkable success conditions, run repeatedly. The verdict was sobering — 55% completion, average 14 steps, and a cost-per-task that made the unit economics shaky. The demos had been the happy path; the suite revealed the distribution.
They also evaluated the trajectory, not just the final answer — did it call the right tools, avoid destructive mistakes, stop when done — which a single run can never show. Armed with real numbers, they fixed the biggest failure modes and tracked completion up to 80% and steps down to 8. The COO's takeaway, adopted as policy: never approve an agent on a demo — ask for completion, steps, and cost over a task suite, and check the path, not just the result.
Tessera assembles 30 representative requests with checkable outcomes and runs the agent over all of them nightly, reporting completion, average steps, and cost, sliced by request type. The first honest completion number — 61% — is exactly what tells them where to invest, and the trajectory review catches a risky shortcut before customers do.
Quiz · Chapter 8
- You can't judge an agent by one run because:
- The numbers that matter are:
- Evaluating the "trajectory" means:
- "We ran it and it looked right" is: