Free lesson · The AI Evaluation Engineer · Lesson 10
Is Something Wrong in Production?
You look after a public chatbot that anyone can use for free. It's been live for six months, and every month you send a short dashboard to the team. This month the numbers moved a lot, and the product lead wants to know by tomorrow whether it's good news, bad news or nothing.
How this lesson works
You'll read a real monthly dashboard and guess what moved it, take the numbers apart by hand, then run a year of traffic through the script to see which alert rules would have fired and why. At the end you'll plan the monitoring for a new product with no hints. Plan on about an hour. The script needs Python with pandas.
The data
WildChat is a free chatbot the Allen Institute for AI ran from April 2023 to April 2024, in exchange for publishing the conversations for research. The 838,000 conversations in the release come with the time, the model version that answered, the language, the country, the number of turns, and a scrambled version of the sender's internet address, so you can count conversations per sender without knowing who they are. You met this data in Lesson 2. This time you'll watch it over a year, the way you would watch your own product.
Source: Zhao, W. et al. (2024), "WildChat: 1M ChatGPT Interaction Logs in the Wild." allenai/WildChat-1M on Hugging Face, ODC-BY license. The script reads only the small columns listed above and skips the conversation text.
Step 1
Read the dashboard
September 2023
October 2023
Conversations
48,642
61,831 (+27%)
Share in Chinese
10.8%
22.8%
Average turns per conversation
1.97
1.87
Traffic is up by a quarter, Chinese has doubled, and conversations got a little shorter.
Predict first
What most likely doubled the Chinese share?
September had 15,884 different senders. October had about senders
October had 15,754 senders, slightly fewer than September. Traffic grew by 13,000 conversations while the number of people stayed flat, so a few senders must have sent a lot more. Five senders sent 1,000 or more conversations each that month. Two of them, both sending only Chinese, sent 2,738 and 2,719. The biggest, in France, sent 6,528, and 6,468 of those arrived on a single day.
This is the first thing to check whenever a traffic number moves: is it many people doing a bit more, or a few doing a lot more? An average over conversations lets one busy sender speak for thousands of real users.
Step 2
Take the month apart
Split October into the five heavy senders (1,000+ conversations each) and everyone else:
Senders
Conversations
In Chinese
October, 5 heavy senders
5
15,909
7,293
October, everyone else
15,749
45,922
6,826
September, everyone else
15,882
45,451
5,240
Your turn
Share of October's conversations sent by the 5 heavy senders: %
Chinese share in October, everyone else: %
Change in conversations from everyone else, September to October: %
Heavy senders: 15,909 of 61,831, 26%. Chinese among everyone else: 6,826 of 45,922, 14.9%. Everyone else grew from 45,451 to 45,922, +1%.
So the 27% growth was five senders. The real user base was flat. Chinese did rise among ordinary users too, from 11.5% (5,240 of 45,451) to 14.9%, a real but much smaller change that the dashboard's 22.8% buried. A team reading the dashboard might have hired a Chinese-speaking support person or celebrated growth. Neither was justified.
The first run reads only the columns it needs from the 14 data files, which takes a few minutes. The first command prints every month with and without heavy senders. The second runs four simple alert rules on every day of the year, each comparing a day with the median of the seven days before it:
rule days it fires of those, one sender sent 5%+
volume below half of usual 18 15
volume above twice usual 5 4
Chinese share moves 5+ points in a day 52 34
average turns 20%+ below usual 12 12
And the third shows the busiest senders on 29 October:
sender 25fe0280 6,468 conversations (85%) English France turns 1.01 08:15 to 18:11
Explain it
The "average turns" alert fired on 12 days, and on all 12 one sender made up at least 5% of the day's traffic. What would you change about how that number is computed?
Sender 25fe0280 sent 6,468 conversations in ten hours, almost all a single message long. What is it most likely, and what would you do about it?
1. Stop letting one sender set the average. Compute quality and behavior numbers per sender first, or count each sender at most a few conversations a day, or leave out senders above a threshold and report them separately. Then add a number the dashboard never had: share of traffic from the busiest sender. If that moves, every other number on the page needs a second look.
2. It's a program. About 11 conversations a minute for ten hours, each a single message, is someone using the free chatbot as a free API, probably to run a batch job. For the product it's a cost and a capacity risk, so it calls for a rate limit per sender. For evaluation it's noise: those 6,468 conversations tell you nothing about how real people use the product, and they'd swamp any sample you draw for review. Lesson 2 found the same pattern as bulk templates in April's traffic.
Step 4
The drop nobody announced
In the summer, average conversation length fell and never came back. The model changed around then too: on 28 June the service switched from the March version of GPT-3.5 to the June version, and GPT-4 had been removed a few days earlier. The product lead's first guess: "the new model is worse, so people give up sooner."
Predict first
When you look week by week, what will you find?
Run python monitor_lesson.py weekly 2023-05-15 2023-08-13. The fifth column leaves out anyone who sent 50 or more conversations that week:
Among ordinary senders, turns were about 3.1 in the two weeks before the switch, 3.08 in the week of the switch and 3.01 the week after, then fell to 2.24 and 1.98. The drop is real, it isn't the heavy senders, and it doesn't line up with the switch. Note also how much the number wanders with nothing changing: 2.49 one week and 3.11 the next in early June. A week-to-week move of 0.3 turns is normal here.
What caused it? This data can't say, and that's the honest answer to give the product lead, along with what you ruled out. The next places to look are outside the dashboard: the app's own release notes for mid-July (a new interface, a change in how conversations are saved or split), and a sample of conversations read side by side from the first and third weeks of July. This is why a monitoring log should record every change to the product, model versions included: when a number moves, the first question is what else changed that week.
Think it through
Besides the model version, what would you want recorded with every conversation so that next time you could explain a drop like this in an hour?
Step 5
Which alert wakes someone up?
The team agrees to page the on-call engineer at night for one rule only. Everything else goes into a morning report. From the alert table in Step 3:
Volume below half of usual: fired on 18 days. Two of them were 25 and 26 June, just before the model switch, when the service answered 529 and then 64 conversations instead of about 3,400. Four were 29 December to 1 January.
Volume above twice usual: fired on 5 days, 4 of them driven by one sender.
Chinese share moves 5+ points in a day: fired on 52 days, 34 of them with one sender at 5% or more of traffic.
Average turns 20%+ below usual: fired on 12 days, every one of them with one sender at 5% or more.
Decide
Volume below half of usual. It's the only one where the likely cause is "the product is broken for everyone," and it caught the two days in June when the service nearly stopped. It needs tuning before it pages anyone: compare with the same weekday and account for holidays, so New Year's Eve doesn't wake someone. Even better, page on a direct signal of failure, like the share of requests that return an error, and keep volume as the backup.
The other three mostly measure one sender's behavior, which is rarely an emergency. The volume spike may deserve a same-day look for cost reasons. The language and turns alerts belong in the morning report, computed without heavy senders, with the busiest sender's share shown next to them.
A useful test for any alert: when it fires, what will the person do? If the answer is "look, and usually find one busy sender," it isn't a page.
Step 6
Plan the monitoring, no hints
A food delivery app is launching an AI assistant in three cities. It answers questions about orders, handles "where is my food" and issues refunds for missing items up to $30 through a tool. You're asked what to monitor from launch day.
Your plan
What would you log, which numbers would go in the daily report and how would you slice them, what would page someone, and how would you check the quality of the answers themselves?
Log: with every conversation, the model version, prompt version, app release, city, language, a scrambled customer id, every tool call and its result, and how the conversation ended (resolved, handed to a person, customer left). Keep a change log of every release and prompt edit with dates.
Daily report: conversations and customers; refunds issued, their total and average amount; share handed to a person; share where the customer came back within a day about the same order; errors and response time. Next to each, the busiest customer's share of traffic and of refunds.
Slices: by city, language and type of request, because the average will hide a problem in one city. Compute rates per customer so one heavy user can't move them.
Page on: error rate or volume collapsing (the service is broken), and refund totals far above normal in an hour (money is leaving). Everything else waits for the morning.
Quality checks: every week, a person reads a sample drawn across cities and request types, sized so a real drop would show (Lesson 1 shows how to size it), and scores it against a short rubric. Check a sample of refunds against the order records, since the conversation can sound right while the refund is wrong. When any number moves, first check the change log and the busiest customers, then read conversations.
Step 7
How your judgment did
What to take away
When a production number moves, ask first whether it's many people or a few. Show the busiest sender's share next to every metric, and compute rates per user so one program can't speak for thousands of people. Expect numbers to wander week to week, and know how much before calling a change. Log every change to the product so a drop can be traced to its week. And page people only for rules where the answer to "what will they do?" is "fix something now."
Data: WildChat-1M, Allen Institute for AI, ODC-BY license. Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., Deng, Y. (2024). WildChat: 1M ChatGPT Interaction Logs in the Wild. ICLR 2024.