Free lesson · The AI Evaluation Engineer · Lesson 11
Ship or Hold?
Your company's support agent runs on GPT-4o and handles two businesses: an online store and an airline. A newer model, Claude 3.5 Sonnet, scores higher on the benchmark your team trusts. The head of support wants to switch next week and asks you for a one-page recommendation: ship the new model, hold it, or something in between.
How this lesson works
This is the last lesson, and it uses what the others taught: paired comparisons and ranges (Lesson 1), reading what went wrong instead of trusting the score (Lessons 4 and 6), consistency across repeated runs and rules in tools (Lesson 8). You'll make the call the way you would at work, then write the memo. Plan on about an hour. The script needs only Python.
The data
The same τ-bench runs as Lesson 8: 115 store tasks and 50 airline tasks, each a simulated customer with a request, graded by whether the database ends up exactly right. GPT-4o ran each task 4 times and Claude 3.5 Sonnet 8 times. Here GPT-4o is "old" and Claude is "new."
Source: Yao et al., 2024, "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." Runs from github.com/sierra-research/tau-bench, MIT license.
Step 1
Is the gain real?
The vendor's summary:
Old (GPT-4o)
New (Claude 3.5 Sonnet)
Store, success per run
60%
69%
Airline, success per run
42%
46%
Predict first
Once you allow for chance, the new model is clearly better in:
domain tasks old, per run new, per run new minus old 95% range resamples new ahead
retail 115 60% 69% +8.8% +2.5% to +15.2% 100%
airline 50 42% 46% +4.0% -5.2% to +13.8% 79%
This is the paired resampling from Lesson 1, with tasks in place of questions: take the difference on each task, then redraw the tasks 5,000 times. In the store the whole range is above zero, so the gain is real. In the airline the range runs from 5 points worse to 14 points better, and in 1 of 5 redraws the old model comes out ahead. With 50 tasks, a 4-point gap can't be told apart from chance.
Step 2
Better on average, better for whom?
Six store tasks, with how many runs each model got right:
Task
What the customer wants
Old (of 4)
New (of 8)
3
Change the small t-shirts in a pending order to purple
4
2
60
Change earbuds in a pending order to blue, same price or lower
4
3
27
Return two items and exchange boots for the waterproof version
3
1
6
Exchange a water bottle and a desk lamp, then confirm only the lamp
0
5
19
Return a bottle and exchange two items for the cheapest versions
1
8
28
Return five items, cancel one new item, ask for the refund total
0
8
Your turn
Old model's average success rate over the six tasks: %
New model's average success rate over the six tasks: %
Number of tasks where the new model's rate is lower: of 6
The average went up, and half these tasks got worse, some by a lot. Across all 115 store tasks the pattern holds: on the 20 tasks that need a single change to a pending order, the new model is no better (67% against 69%). Its gains come from requests with several parts: tasks needing two order changes go from 25% to 71%. If most of your real tickets are simple one-change requests, the headline gain mostly won't reach your customers. An average across tasks is only as good as the match between the tasks and your traffic, which is Lesson 2's question again.
The first command repeats Step 1 and adds success on all 4 runs. The second lists the store tasks where the new model is much worse:
retail: old model right on every run for 44 tasks; new model fails at least once on 20 of them.
Tasks where the new model's success rate is 50+ points lower:
task 3 old 4/4 new 2/8 required: modify_pending_order_items
task 27 old 3/4 new 1/8 required: exchange_delivered_order_items
task 45 old 4/4 new 4/8 required: exchange_delivered_order_items
task 49 old 3/4 new 2/8 required: exchange_delivered_order_items
task 60 old 4/4 new 3/8 required: modify_pending_order_items
task 66 old 3/4 new 0/8 required: cancel_pending_order
task 92 old 3/4 new 0/8 required: return_delivered_order_items, return_delivered_order_items
task 94 old 4/4 new 3/8 required: exchange_delivered_order_items
Tasks where it is 50+ points higher: 18
Task 66 is the worst: 3 of 4 for the old model, 0 of 8 for the new. Read one of each, and a second new run:
python release_lesson.py show retail gpt-4o 66 0
python release_lesson.py show retail sonnet 66 0
python release_lesson.py show retail sonnet 66 1
Explain it
Did the new model make a mistake in task 66? If not, what failed?
The old model was right on every run for 44 tasks, and the new one slips on 20 of them. What does that mean for customers on launch day, and what would you do about it?
1. Mostly the test failed. The customer wants to swap a luggage set for a coat, and the policy forbids swapping between product types, so the only option is to cancel the whole order. The grader expects a cancellation with the reason "no longer needed." In three of the new model's runs the simulated customer said it was "ordered by mistake," and the agent cancelled with that reason, so the database doesn't match. In the other five, the simulated customer learned the order held other items too and decided to keep it, which the task's instructions don't allow for. None of those runs is a clear model error. In the old model's three successful runs, the cancellation reason happened to match the one the grader wanted.
This is the regression that would have held the release, and it's largely noise from the benchmark. Before you hold anything for a regression, read it.
2. Some customers who were served well will now be served badly. Even with the average up, people whose requests look like tasks 3 and 60 will notice. Read each large regression and sort it: model error, test problem, or ambiguous request. Fix the tests that are wrong. For real model errors, see whether a prompt or tool change fixes them before launch. Then launch in stages, a share of traffic at a time, and watch simple one-change requests separately from the rest.
Step 4
The failures that cost money
Not all failures cost the same. A missed exchange annoys a customer. A refund or cancellation the policy doesn't allow costs the company money. In Lesson 8, that was the most common airline failure for GPT-4o: 23.5% of all its airline runs made a change they shouldn't have.
Predict first
For the new model, the share of airline runs that make a change they shouldn't is:
Run python release_lesson.py failures. These are shares of all runs, so the two models can be compared directly:
retail old new
made a change it should not have 5.7% 4.0%
right kind of change, wrong details 10.0% 9.8%
skipped a required change 14.6% 8.4%
made no change 7.6% 5.4%
changes right, reply to the customer wrong 1.7% 3.2%
airline old new
made a change it should not have 23.5% 23.8%
right kind of change, wrong details 7.5% 9.8%
skipped a required change 12.0% 9.2%
made no change 13.5% 10.8%
changes right, reply to the customer wrong 1.5% 0.5%
In the airline, nearly one run in four still makes a change the policy forbids, with either model. The newer model gets things done more often, but it isn't more careful with the rules. That's the pattern Lesson 8 predicted: while the policy lives only in the prompt, a model's judgment in the moment decides, and switching models doesn't change that. The fix is in the tools. In the store, the new model is better on the costly failures too, from 5.7% of runs to 4.0%.
Note that these are rough labels from comparing database changes, as in Lesson 8, so read a few before quoting them. The size of the airline problem is large enough that the labels would have to be very wrong to change the conclusion.
Think it through
The head of support hears "success went from 42% to 46%" and "harmful changes stayed at 24%." Which number should drive the airline decision, and why?
Step 5
Make the call
Decide
Switch the store in stages. Fix the airline's tools first. In the store the gain is real (+9 points, range +2 to +15), costly failures went down, and the biggest regression is mostly the test's fault. Staging protects against the regressions that are real: start with a share of traffic and watch simple one-change requests on their own.
In the airline, the evidence doesn't say the new model is better, and the problem that costs money, forbidden changes in a quarter of runs, is the same with either model. No model switch fixes that. Moving the cancellation and refund rules into the tools does, as Lesson 8 showed. After that fix, rerun both models on the airline tasks with more runs per task, because 50 tasks can't separate a 4-point gap.
Holding everything until the new model wins on every task sounds careful, but no model change ever meets that bar, so in practice it means never switching. Switching only the airline puts the change where the evidence is weakest.
Step 6
Write the memo, no hints
The head of support will read one page and decide. Write it: the recommendation, the evidence, the risks, and what happens next.
Your memo
Recommendation. Move the store's support agent to Claude 3.5 Sonnet in stages, starting next week. Keep the airline on its current model for now and first change the booking tools so they enforce the cancellation and refund rules.
Evidence. Store: over 115 benchmark tasks, success per run rises from 60% to 69%, and the likely range of the gain is 2 to 15 points. The share of runs that make a change the policy forbids falls from 5.7% to 4.0%. Airline: success rises from 42% to 46%, but over 50 tasks that gap could be chance (range −5 to +14). Forbidden changes stay at about 24% of runs with either model.
Risks. The gain comes from requests with several parts. On simple one-change requests the new model is no better (67% against 69%), and on some tasks the old model handled every time it slips. We read the worst case and found it was mostly a flaw in the test. Others may be real. The benchmark's customers are simulated, so our real ticket mix may differ.
Next steps. (1) Read the eight large store regressions and fix any test errors by the end of the week. (2) Start with 10% of store traffic for a week. Compare resolution rate, refunds and hand-offs to people with the old model, with simple one-change requests reported on their own, then increase. (3) Engineering adds rule checks to the airline cancellation and refund tools. (4) Rerun both models on airline tasks with 8 runs each, then decide the airline.
Notice what the memo leaves out: no "the new model is smarter," no single headline number, and every claim comes with its range or its caveat. The reader can check each one.
Step 7
How your judgment did
The questions to carry into every evaluation
Is the difference bigger than chance? Compare on the same items, and give a range. (Lesson 1)
Does the test set look like the traffic? (Lesson 2)
Does the score agree with people, and whose? (Lessons 3 to 5)
How often does it make things up, and what would the checker catch? (Lesson 6)
Did the right information reach the model at all? (Lesson 7)
Does it work every time, or only sometimes? Which rules live in code? (Lesson 8)
Safe against which attacks, judged by whom, and at what cost in refusals? (Lesson 9)
After launch: many people or a few, and what else changed that week? (Lesson 10)
Before any decision: read the failures behind the number. (all of them)
Data: τ-bench historical trajectories, Sierra, MIT license. Yao, S. et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.