Why Agents Are a Different Risk
"Agents" — AI that takes multiple steps and uses tools (sends email, queries databases, takes actions) — are the most exciting and most dangerous thing your team can build, and the hardest to evaluate.
With a chatbot, you judge one answer. With an agent, you must judge a whole sequence of actions. Two things make this risky for the business. First, an agent can reach the right outcome the wrong way — completing the task while taking a destructive or expensive detour — so you have to evaluate how it acted, not just whether it finished. Second, errors compound: an agent that's 95% reliable per step is only about 60% reliable over ten steps and a coin flip by twenty. And because agents act, cost and latency are quality issues — a correct agent that costs $4 and takes five minutes per task may be unshippable.
| Steps in the task | Per-step reliability | Chance the whole task succeeds |
|---|---|---|
| 1 | 95% | 95% |
| 10 | 95% | ~60% |
| 20 | 95% | ~36% |
The agent that "succeeds"
A team demos an agent that resolves support tickets end-to-end with an 80% success rate and wants to give it the power to issue refunds. What two things must you see before you grant that power?
One: trajectory evaluation, not just outcome — of the runs that "succeeded," how many took a safe path? An agent that resolves tickets by over-refunding is "successful" and a liability. Two: a safety gate on the irreversible action — refunds above a threshold should require confirmation, and you'll want the failure cost modeled (20% failure on a refund agent isn't a quality stat, it's money out the door). Pair that with cost/latency per ticket. "80% success" alone is nowhere near enough to hand over the company checkbook.
Ledgerline: the refund that shouldn't have fired
Ledgerline's billing agent could issue refunds on its own — convenient, until a malformed ticket and a confused loop led it to refund a customer $4,000 that was never owed. The action was instant and irreversible; unwinding it took a day of finance work and an awkward call. The root problem wasn't a bad model — it was an autonomous system wired directly to a high-impact, unrecoverable action, evaluated only on whether it "resolved the ticket."
Evaluating an agent means checking the actions and the path, not just the final state, and measuring a harmful-action rate — not only a completion rate. Ledgerline added a human approval gate on refunds above a threshold, which also stopped a malicious ticket from triggering a payout. The leadership lesson: once your AI can act, "did it finish?" is not enough — you must measure "did it ever do something you can't take back?"
This chapter: Remi now issues refunds, not just answers about them. Over a 200-ticket evaluation it resolves 82%, but the number that stops Meridian is a 3% wrong-amount-refund rate — invisible if you only track "resolved," visible because they scored the action taken. They gate refunds above a threshold behind a human. (Ch 9: they try to break Remi on purpose.)
Quiz · Chapter 8
- Evaluating an agent is harder than a chatbot because you must judge:
- "Errors compound" means a 95%-per-step agent over 20 steps is:
- Before giving an agent power over an irreversible action (refunds, deletes), you need:
- For agents, cost and latency are: