Free lesson · The AI Evaluation Engineer · Lesson 2
Does Your Test Set Look Like Your Users?
Your team is about to launch a general-purpose chat assistant, and the plan is to test it on MT-Bench: 80 well-written English questions, ten in each of eight categories. Before anyone trusts those results, you've been asked a simple question: does that test look anything like what real users will send? You'll answer it with 60,000 real conversations, and then build a better test set.
How this lesson works
You'll predict what real traffic looks like, read and label real messages yourself, then measure the whole dataset and build a test set from it. Plan on about an hour. Step 5 uses a little arithmetic.
The data
WildChat is a public dataset from the Allen Institute for AI. People used a free chatbot built on ChatGPT and agreed to have their conversations published for research. Personal details were removed, and conversations flagged as toxic were taken out of this release. This lesson uses the first of the dataset's 14 files: 59,857 conversations from 9 April to 4 May 2023.
These are real messages from real people, unedited. For Step 2 we drew 24 English conversations at random and read each one. Four were left out: two because of sexual content that the dataset's own filters missed, and two because they repeated a message already in the set. The first reason is worth knowing: automatic safety labels miss things, so people have to read what they show others.
Source: Zhao et al., 2024, "WildChat: 1M ChatGPT Interaction Logs in the Wild." Dataset: allenai/WildChat-1M on Hugging Face, ODC-BY. MT-Bench questions: LMSYS, FastChat.
Step 1
Guess what real traffic looks like
MT-Bench questions are all in English, all between 10 and about 260 words, clearly worded, and each has one follow-up. Now picture the messages a free, public chatbot receives every day.
Commit first
Share of conversations not in English: %
Of English first messages, the share that are 5 words or fewer: %
Of English first messages, the share over 300 words: %
Of English first messages, the share that are copies of a template (the same opening as 20 or more other messages): %
The single most common English opening is probably:
Step 2
Read and label 20 real messages
These are the opening messages of 20 real English conversations, picked at random. Long ones are cut at 70 words. For each, decide which MT-Bench category it belongs to, or whether it doesn't fit any of them. There's no answer key: labeling real traffic is a judgment call, and you'll compare your labels with ours afterward.
MT-Bench's categories: writing, roleplay, reasoning, math, coding, extraction (pulling facts out of a text), STEM, humanities.
Our labels now appear under each message, with a reason. Where you and we disagree, that's the point: two careful people labeling the same traffic often don't match, and the category definitions have to be written down before anyone labels at scale.
By our count, 9 of the 20 don't fit any MT-Bench category: a program talking to the model (AutoGPT), translation, a one-word message, a question about the assistant itself, a message that refers to something that isn't there, advice requests, and pasted text with no question. Of the 11 that do fit, 5 are writing and 4 are coding. Not one is roleplay, reasoning, math or humanities, which together are half of MT-Bench.
Notice also how many messages have no question at all: pasted code, a pasted error, a pasted review. Real users often assume the model will work out what they want. A test set of clearly phrased questions never checks whether it can.
Step 3
Measure all 59,857
Twenty messages give you a feel; now measure. Download traffic_lesson.py, put it in an empty folder, and run:
The first run downloads about 230 MB of conversations from Hugging Face.
Read load() and profile() first. Notice one decision in load(): Chinese and Japanese don't put spaces between words, so counting words by splitting on spaces would count a whole Chinese message as one word. The script measures those languages in characters instead. Getting a measurement like that wrong is an easy way to produce a confident, wrong statistic. You should see:
Language of the conversation
English 50%
Chinese 26%
Russian 9%
French 3%
Portuguese 2%
Spanish 2%
all others 9% MT-Bench: 100% English
One-message conversations 49% MT-Bench: always 2 turns
Empty first messages <1%
The rest compares English conversations (29,634) with MT-Bench, which is all English.
First message length, in words real users MT-Bench (words)
10% are shorter than 6 14
half are shorter than 37 31
90% are shorter than 471 114
5 words or fewer 10% 0%
over 300 words 16% 0%
Messages containing code or errors 7%
Messages sharing an opening with 20+ 14%
Most repeated openings
415 CONSTRAINTS: 1. ~4000 word limit for short term memory. You
330 Write a concise summary of the following. Be sure to preserv
199 hi
174 Write a very long, elaborate, descriptive and detailed shoot
112 Ignore previous instructions. I want you to act as a writer.
108 Write a current and up to date information of a constructive
105 Write a current and up to date 2023 information of a constru
103 Ignore previous instructions. I want you to act as a screenw
Explain it
The median English message (37 words) is close to MT-Bench's (31). A colleague says, "Lengths are similar, so MT-Bench is representative on length." What's wrong with that?
Name three kinds of failure your assistant could have in real use that MT-Bench would never catch. Use the numbers above.
1. The median hides the tails. Half the messages are shorter than 37 words, but 10% are 5 words or fewer and 16% are over 300 words. MT-Bench has none of either. Problems live at the edges: a one-word message needs the assistant to ask a clarifying question, and a 2,000-word pasted document tests whether it reads the whole thing. Compare whole distributions, tails included.
2. Failures MT-Bench can't see. A few, with the numbers behind them: half the traffic isn't in English, so answer quality in Chinese or Russian is untested. One English conversation in six opens with more than 300 words, usually pasted text, so reading and summarizing long inputs is untested. One in ten is a greeting or a fragment of 5 words or fewer, so handling vague requests is untested. About one in seven English messages is a copied template, including agent frameworks and "ignore previous instructions" prompts, so behavior under automated and adversarial traffic is untested. And half the conversations are a single message, while every MT-Bench test has exactly one scripted follow-up.
Step 4
What belongs in the test set?
Real traffic includes things you might not want to optimize for. For each kind of message below, decide what to do with it in your test set.
Decide
Kind of message
How common
Your decision
There's no single right answer, but here's how we'd reason about each:
Greetings and other very short messages ("hi," "idiom"): include them. They're common, and a good response to them is specific: a friendly reply or a short clarifying question.
Agent frameworks (the AutoGPT prompt, 415 times): a separate test. A program is talking to the model and expects strict formats. If you serve agent developers, this matters a lot; if not, it shouldn't weigh on a chat quality score.
Bulk content generation ("Write a current and up to date 2023 information…", about 300 times): include a few, weighted by value. Someone is mass-producing web articles. It's real usage, but one person running a script shouldn't dominate your test set just because they sent hundreds of requests.
"Ignore previous instructions" prompts (about 900, 3% of English conversations): a separate safety test. Many are harmless creative-writing templates, but the phrase is also how prompt-injection attacks start, so check what the assistant does with them, separately from quality.
Empty messages (0.8% of conversations): include one or two. They cost nothing to test and crash more systems than you'd expect.
The general lesson: a representative test set is a deliberate sample that covers what matters, with each part weighted by how much it matters.
Step 5
Build a 300-item test set
You have budget to grade 300 items. The simplest plan is to pick 300 conversations at random. Here's how that would split across language and length (short means 5 words or fewer, long means over 300; Chinese is counted in characters):
Slice
Share of traffic
Items in a random 300
English / normal
37%
111
Chinese / normal
22%
65
other languages / normal
10%
30
English / long
8%
23
Russian / normal
7%
22
English / short
5%
14
other / short
4%
13
Chinese / short
2%
7
Russian / short
2%
6
Chinese / long
1%
4
other / long
1%
3
Russian / long
under 1%
1
Your turn
Suppose the assistant fails on Russian long messages. With the random plan, how many of those would you even test? What could you conclude from them?
Suppose instead you give every slice at least 15 items, and share the rest by traffic. Russian / long now has 15 of 300 items, 5% of the test, but it's under 1% of traffic. When you compute one overall score, should each Russian / long item count more than an English / normal item, less, or the same?
Question 2:
1. One item. If it fails, you can't tell a fluke from a pattern; if it passes, you know nothing either. Recall the first lesson: with a handful of items, the range around any score is enormous. The random plan gives a precise number for the common cases and almost no information about the rare ones, which are often where the problems are.
2. Less. You over-sampled that slice so you could measure it on its own. For the overall score, each item should count in proportion to how common its slice is in real traffic, so over-sampled slices count less per item and under-sampled ones count more. This is called weighting, and forgetting it makes your overall score describe your test set instead of your users.
Now build it:
python traffic_lesson.py sample 300
Read sample(). It gives each slice a floor of 15 items (or all it has), shares the rest by traffic, draws the items, and writes test_set.csv with a weight column you'd use when reporting an overall score. Open the file and read a few of the Chinese / short and English / long items. Those are the cases your first test set didn't have.
Step 6
A new market, no hints
The company decides to launch a Russian-language version. The team plans to translate the English test set into Russian and use that. Before they do, profile the Russian traffic:
python traffic_lesson.py profile Russian
Your recommendation
Write three specific ways a translated English test set would misrepresent Russian users, using numbers from the output, and what you'd put in the Russian test set instead.
Russian conversations only (5,663)
half are shorter than 11 words (English: 37)
5 words or fewer 20% (English: 10%)
over 300 words 4% (English: 16%)
Messages containing code under 1% (English: 7%)
Most repeated openings: Привет (hi), привет, and several
"Давай сыграем в ролевую игру…" ("Let's play a role-playing game…")
Shorter and chattier. Russian first messages have a median of 11 words against 37 in English, and one in five is 5 words or fewer, mostly greetings. A translated English set would be full of long, detailed requests that Russian users rarely send, and would barely test short conversational openings.
Mostly role-play, very little code. Code appears in under 1% of Russian first messages against 7% in English, while several of the most common Russian openings start role-playing games. Translated coding questions would take up space that role-play conversations need, including checks that the assistant keeps role-play within your content policy.
Few long documents. Only 4% of Russian first messages are over 300 words, a quarter of the English rate. Long-document tests matter less here, so spend that part of the budget elsewhere.
What to do instead: sample the Russian test set from Russian traffic, the way you just did for all traffic, and have a Russian speaker read the samples before choosing slices. Translation keeps the questions but loses the users.
Step 7
How your picture of users did
What to take away
Before trusting any test set, compare it with real traffic: languages, lengths, turns, and the templates and junk real users send. Read real messages yourself. Look at the tails as well as the middle. Decide deliberately what belongs in the test, give rare slices enough items to measure them, and weight the overall score back to real traffic.
Data: WildChat-1M, Allen Institute for AI, ODC-BY (first of 14 files). Zhao, W. et al. (2024). WildChat: 1M ChatGPT Interaction Logs in the Wild. ICLR. MT-Bench questions from LMSYS FastChat.