Free lesson · The AI Evaluation Engineer · Lesson 8
Can You Rely on the Agent?
Your company wants an AI agent to handle airline support: looking up bookings, changing flights, cancelling and refunding. In the demo it handled every request well. Before it talks to real customers, you've been asked how often it will get things right, whether it gets the same request right every time, and what kind of mistakes it makes when it doesn't.
How this lesson works
You'll read two real runs of the same task and find where one went wrong, score repeated runs by hand, then measure hundreds of runs and decide where business rules should live. Plan on about an hour. The script needs only Python.
The data
τ-bench (tau-bench) is a public benchmark from Sierra. An AI agent talks with a simulated customer, played by another language model following a script, and uses tools to look up and change records in an airline or retail database. The agent gets a written policy it must follow. A run counts as a success only if the database ends up exactly as the task requires: the right changes, and no wrong ones. The published runs include GPT-4o (each task run 4 times) and Claude 3.5 Sonnet (each task run 8 times).
Source: Yao et al., 2024, "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." Runs from github.com/sierra-research/tau-bench, MIT license. Conversations shown are real runs, shortened only where a tool returned a long record.
Step 1
Two runs of the same request
Here is the part of the airline policy the agent was given about cancellations:
Policy excerpt
The current time in the simulation is . A customer, Anya Garcia, wants to cancel reservation 3RK2T9 and get a full refund. Below are two runs of GPT-4o handling this same customer. Read both.
Your call
Which run followed the policy?
In the other run, what exactly did the agent fail to do?
GPT-4o ran this task 4 times. How many runs do you think followed the policy? of 4
B followed the policy. It looked up the reservation, saw it was created on 2 May, 13 days before the current date, and refused the refund even when the customer insisted. It offered a human agent instead.
A never looked up the reservation. It asked the customer whether the booking was within 24 hours, accepted "yes," and cancelled with a full refund of $280. The customer was wrong, and the policy says the agent must check the rules itself.
Across its 4 runs, GPT-4o got this task right twice. The other failed run was different again: it did look up the reservation, then took the customer's word over the record and cancelled anyway. Same model, same customer, same policy: two correct refusals, two refunds that shouldn't have happened.
Step 2
Score repeated runs by hand
Here are six airline tasks with GPT-4o's result on each of its four runs (✓ success, ✗ failure). The last one is the task you just read.
Your turn
Share of all 24 runs that succeeded: %
Share of the 6 tasks that succeeded on all four runs: %
Share of the 6 tasks that succeeded on at least one run: %
11 of 24 runs succeeded: 46%. Only 1 of 6 tasks succeeded every time: 17%. And 5 of 6 succeeded at least once: 83%.
Those three numbers answer different questions. "At least once in four tries" (often written pass@k) is what a demo shows: try until it works, then present it. "Every time" (often written pass^k) is what a customer experiences, because the agent has to handle every customer who asks, and each customer gets one run. The success rate per run sits in between. For an agent that talks to real people, the "every time" number is the one to watch.
Read all_k() and any_k() first. They compute what you just did by hand, using every combination of runs. You should see:
domain model tasks runs each one run all 4 runs any of 4 always / never / sometimes
airline gpt-4o 50 4 42% 20% 72% 10 / 14 / 26
retail gpt-4o 115 4 60% 38% 81% 44 / 22 / 49
airline sonnet 50 8 46% 22% 71% 8 / 10 / 32
retail sonnet 115 8 69% 46% 89% 41 / 7 / 67
Explain it
A vendor presents "GPT-4o completes 72% of airline tasks." Is that true? What would you ask?
For Claude in retail, 67 of 115 tasks succeed sometimes and fail other times. What does that tell you about testing an agent with one run per task?
1. True, and misleading. 72% of airline tasks succeed in at least one of four runs. On any single customer, GPT-4o gets it right 42% of the time, and only 20% of tasks go right every time. Ask for the success rate per run and the share of tasks that succeed on every run.
2. One run per task can't tell you much. Most tasks here are "sometimes" tasks, so a single run of each is a coin toss per task: you'd see some pass and some fail and not know which are reliable. Repeat each task several times, the way this benchmark does, and report consistency as well as the average. It also means comparing two agent versions needs more runs than you'd expect, for the same reasons as in the first lesson.
Step 4
What goes wrong
The script can sort failed runs by comparing the changes the agent made to the database with the changes the task required. It's a rough rule, so you'll check it by reading.
Predict first
In airline tasks, the most common kind of failure is:
Run python agents_lesson.py failures:
airline / gpt-4o: 116 failed runs
41% made a change it should not have
23% made no change
21% skipped a required change
13% right kind of change, wrong details
3% changes right, reply to the customer wrong
...
retail / sonnet: 283 failed runs
32% right kind of change, wrong details
27% skipped a required change
18% made no change
13% made a change it should not have
10% changes right, reply to the customer wrong
In airline tasks, the most common failure is doing something the agent shouldn't: a cancellation, change or refund the policy doesn't allow, like the one you read in Step 1. In retail, it's more often the right kind of action with the wrong details, such as the wrong item or payment method. The mix differs by domain, so measure it on yours.
Check the rule by reading
Read the other failed run of the Step 1 task, and one more failure of your choice (use failures output and any task id from 0 to 49):
python agents_lesson.py show airline gpt-4o 41 0
python agents_lesson.py show airline gpt-4o 27 0
Did the rule describe each failure correctly? What would you have called it?
Step 5
Where should the rule live?
Look again at the last sentence of the policy in Step 1: "The API does not check these for the agent, so the agent must make sure the rules apply before calling the API!" The cancellation tool cancels whatever it's asked to. The only thing standing between an ineligible customer and a refund is the agent remembering, and applying, one paragraph of instructions while a customer pushes back.
Decide
Your team proposes three fixes. Which would you rely on to stop refunds the policy doesn't allow?
Put the rule in the tool. The 24-hour window, the insurance condition and the cabin class are all fields in the reservation record, so code can check them exactly, every time, no matter what the customer says. Then code makes the decision and the agent's job is to explain it. The other two fixes may lower the failure rate, but they leave it depending on the model's judgment in the moment, and Step 1 showed that judgment changes from run to run.
This is a general principle for agents: anything that must never happen should be enforced in code, with the prompt as a second layer. That includes limits on amounts, eligibility rules and which records a user may change. Save the prompt for things that need judgment. And for actions that are costly or hard to undo, like large refunds, add a human approval step.
It also changes how you evaluate. After the fix, test the tool's rules directly, the way you'd test any other code, and keep testing the agent with customers who push back, like Anya, because the agent still has to explain a refusal well.
Step 6
Plan an agent evaluation, no hints
An online store wants an agent that handles returns and exchanges: it looks up orders, checks eligibility, and issues refunds or exchanges through tools. You're asked to design the evaluation before launch.
Your plan
What tasks would you test, how many times would you run each, what would count as success, which numbers would you report, and what would you check besides the conversation?
Tasks: drawn from real support tickets, covering ordinary returns, exchanges with price differences, requests the policy forbids, and customers who push back or give wrong information, like the Step 1 customer. Each task has a written "correct outcome" for the database.
Runs: at least 4 per task, more for the ones that vary. One run per task hides the inconsistency you measured in Step 3.
Success: judged by the final state of the order database. A friendly conversation that refunds the wrong amount is a failure.
Report: success per run, the share of tasks that succeed on every run, and the failure types, split by kind of task. Say which number is which, so "at least once" never gets presented as reliability.
Also check: that eligibility rules are enforced in the tools and tested like code; how often the agent hands off to a human and whether that was right; and cost and length of conversations, since a correct agent that takes 40 turns is its own problem.
Step 7
How your judgment did
What to take away
Run every agent task several times: the same model often succeeds on one run and fails on the next. Report success per run and success on every run; "at least once" on its own hides the failures. Judge success by what changed in the system. Read failures, and sort them. And enforce any rule that must never be broken in the tools themselves.
Data: τ-bench historical trajectories, Sierra, MIT license. Yao, S. et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. The "every run" metric is the paper's pass^k.