Free lesson · The AI Evaluation Engineer · Lesson 1
Is the New Model Better, or Is It Noise?
Your product runs on Llama-2 13B. A teammate tested the 70B version on a standard benchmark and it scored higher: 6.86 against 6.65. The 70B model has more than five times as many parameters, so it costs several times as much to run. Your job is to decide whether that score difference is evidence or noise, and what you'd need to know to be sure.
How this lesson works
You'll commit to a judgment before you see the analysis, work through small numbers by hand, then run the full analysis yourself. You'll probably be surprised at least once. Plan on about an hour, with a calculator or phone for Steps 2 and 3.
The data
MT-Bench is a set of 80 questions in eight categories (writing, roleplay, reasoning, math, coding, extraction, STEM and humanities), each with a follow-up question, so 160 items in all. Researchers at LMSYS ran 34 chatbots on it and had GPT-4 score every answer from 1 to 10. Every model answered the same 160 items. The scores here are exactly as published.
One caution before you start: every score in this lesson comes from GPT-4 acting as a judge. Whether that judge can be trusted is a separate question, covered in the lesson Can You Trust an AI Judge? Here, take the scores as given and ask a different question: even if every score is fair, does the difference mean anything?
Source: Zheng et al., 2023, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." Grades published by LMSYS with FastChat (huggingface.co/spaces/lmsys/mt-bench).
Step 1
Your gut call
Here are the two numbers your teammate brought you, and, while you're at it, part of the MT-Bench leaderboard.
Model
Average score (160 items)
Llama-2 70B chat
6.86
Llama-2 13B chat (what you run today)
6.65
Commit first
Is 70B's 0.21-point lead a real improvement?
How sure are you that 70B would score higher if you tested on more questions like these? %
How likely is it that tulu-30b, first on the leaderboard above, is truly the best of those seven? %
Step 2
Look at the items
An average hides the items behind it. Here are 12 of the 160 items, drawn at random, with both models' scores.
Your turn · count and calculate
Items where 70B scored higher: lower: the same:
Average difference (70B minus 13B) on these 12 items:
70B scored higher on 2 items, lower on 3, and the same on 7. The differences add up to −2.5, so the average is −0.21. On this sample, the smaller model looks better, by the same margin that the full 160 items favor the larger one.
Two things to take from this. First, a sample of 12 flipped the conclusion completely. That's not bad luck in some rare way; you'll see in Step 4 how often it happens. Second, 7 of the 12 items gave both models the same score. Those items tell you nothing about which model is better. The decision rests on the 5 items where they differed, and 5 is very few.
Step 3
How far can chance move the number?
To know whether a difference is real, you need to know how much it could move just from which questions happened to be in the test. That's what a 95% range tells you.
The recipe, for a difference measured on the same items:
Work out the difference for each item (you did).
Take the standard deviation of those differences, a measure of how much they spread out. For your 12 items it's 0.89.
Divide by the square root of the number of items. That's the standard error: how much the average itself would wobble from one sample to the next.
Multiply by about 2 (1.96, to be exact). The average, plus or minus that amount, is the 95% range.
Your turn
For your 12 items, the 95% range is −0.21 plus or minus
For all 160 items, the standard deviation of the differences is 2.08 and the average is +0.21. The 95% range is +0.21 plus or minus
For 12 items: 0.89 ÷ √12 = 0.26, times 1.96 = ±0.50. So the true difference could be anywhere from about −0.71 to +0.29. Your sample couldn't tell you which model is better, and a careful reader of it would have said so.
For 160 items: 2.08 ÷ √160 = 0.16, times 1.96 = ±0.32. So the true difference is somewhere between about −0.11 and +0.53. Zero is inside that range. Even the full benchmark doesn't establish that 70B is better.
Predict first
You just used the differences item by item. Another approach is to treat the two models' 160 scores as two separate lists and compare their averages, ignoring which scores came from the same question. Would that range be wider or narrower than ±0.32, and why?
Wider: ±0.79, more than twice as wide.
Here's why. Questions differ a lot in difficulty. On the median-of-two-arrays coding question in your table, both models scored 2; on the argument-evaluation question, both scored 10. Treated as two separate lists, all that difficulty variation looks like noise and swamps the small difference between the models. Taking the difference on each item cancels out whatever the two models have in common on that question, like how hard it is, and leaves only the part that differs between them.
That's why you should always test two versions on the same questions and analyze the differences item by item. It costs nothing and, here, makes the test as sensitive as one with about six times as many questions analyzed the other way.
Step 4
Run the full analysis
Download noise_lesson.py, put it in an empty folder, and run:
pip install pandas numpy
python noise_lesson.py
The first run downloads the published grades (about 20 MB).
Before looking at the output, read compare() and leaderboard(). The script does what you did by hand, plus one more thing: it resamples. It builds 5,000 new versions of the benchmark by drawing 160 items at random from the real 160, with repeats allowed, and checks the result each time. That shows how often the conclusion would change if the benchmark had been built from a slightly different set of questions. You should see:
Llama-2-70b-chat vs Llama-2-13b-chat (160 items)
average score 6.86 vs 6.65
difference +0.21
95% range, unpaired +/- 0.79
95% range, paired +/- 0.32
items: Llama-2-70b-chat higher 45, lower 30, same 85
resampled 5000 times: Llama-2-70b-chat came out ahead in 89% of resamples
by category (20 items each):
writing +0.45 +/- 0.70
roleplay +0.00 +/- 0.55
reasoning +0.70 +/- 1.45
math -0.15 +/- 0.69
coding +0.15 +/- 0.71
extraction +0.33 +/- 1.47
stem +0.30 +/- 0.74
humanities -0.12 +/- 0.49
The leaderboard, and where each model lands when the questions are resampled
model score rank in 90% of resamples ranked 1st
tulu-30b 6.43 1 to 7 18%
guanaco-65b 6.41 1 to 7 13%
oasst-sft-7-llama-30b 6.41 1 to 7 14%
palm-2-chat-bison-001 6.40 1 to 7 19%
mpt-30b-chat 6.39 1 to 7 17%
vicuna-13b-v1.3 6.39 1 to 7 12%
wizardlm-13b 6.35 1 to 7 6%
Explain it
70B came out ahead in 89% of resamples. Is that enough to pay several times more for it? What would change your answer?
On reasoning, 70B leads by 0.70. A teammate says, "Our product is mostly reasoning questions, so the case for 70B is strong." What do you say?
The leaderboard lists seven models in order. What should it show instead?
1. 89%. It means that in roughly one resampled benchmark in nine, 13B came out ahead or tied. It leans toward 70B, but it's weak evidence for a decision that multiplies your running cost. Whether it's enough depends on what's at stake: for a free experiment, fine; for a permanent cost increase, you'd want more evidence, ideally on your own users' questions. Note also that 85 of the 160 items were ties, so the whole result rests on 75 items.
2. Reasoning +0.70. With only 20 reasoning items, the range is ±1.45, from about −0.75 to +2.15. That's the cost of slicing: each slice has fewer items, so each slice's range is wider. The reasoning number is a hint worth testing on more reasoning questions before anyone treats it as a finding.
3. The leaderboard. Each of the seven models lands anywhere from 1st to 7th across resamples, and none is ranked first more than about one time in five. Tulu-30b is "first" by 0.02 points. An honest leaderboard would show them as tied, for example by grouping models whose ranges overlap or printing each score with its range. Ranking them 1 through 7 presents noise as information.
Step 5
What would settle it?
The range shrinks with the square root of the number of items. To cut it in half you need four times as many items; to cut it to a third, nine times as many.
Your turn
To shrink the paired range from ±0.32 to ±0.10, about how many items would you need, if the new items behave like these?
MT-Bench only has 160 items. Write down two practical ways to get a clear answer about 70B versus 13B for your product.
The range needs to shrink by a factor of 3.2 (0.32 ÷ 0.10), so you need 3.2² ≈ 10 times as many items: about 1,600.
Some practical options: build a test set from your own users' real questions, which you'd want anyway because a public benchmark isn't your traffic; spend the effort on the kinds of questions where the models differ, since tied items carry no information; run both models side by side on a slice of live traffic and compare outcomes you care about; or decide on other grounds. If the quality difference is too small to measure on 160 items, it may be too small for your users to notice, and cost and speed can decide.
For the decision you were given: the evidence doesn't justify switching. Stay on 13B, and if 70B still looks interesting, test it on a larger set built from your own traffic.
Step 6
A new decision, no hints
A different company is choosing between GPT-3.5 Turbo and Claude v1. Their MT-Bench averages are 7.94 and 7.90, and a manager wants to standardize on GPT-3.5 "because it scored higher." The company has two products: a science tutor and a math homework helper.
Then write what you'd tell the manager, for each product. Use at least three numbers from the output.
gpt-3.5-turbo vs claude-v1 (160 items)
difference +0.04
95% range, paired +/- 0.40
items: gpt-3.5-turbo higher 34, lower 50, same 76
resampled 5000 times: gpt-3.5-turbo came out ahead in 58% of resamples
by category: math +1.50 +/- 1.59, stem -1.00 +/- 0.51, writing -0.30 +/- 0.38, ...
The average says nothing. A 0.04 lead with a ±0.40 range, ahead in 58% of resamples, is close to a coin flip. "It scored higher" isn't a reason.
The item counts say something the average hides. Claude scored higher on 50 items and GPT-3.5 on only 34. GPT-3.5's average is propped up by a few large wins, mostly in math, while Claude wins more often by smaller margins. Always look at the count of wins and losses along with the average.
Science tutor: on STEM, Claude leads by 1.00 with a range of ±0.51, so the whole range is in Claude's favor. That's the one clear result here, and it points to Claude for this product. It's still only 20 items, so confirm it on the tutor's own questions before committing.
Math helper: GPT-3.5 leads by 1.50 on math, but the range is ±1.59, which just includes zero. It's a strong hint that still needs confirming. Test both on a few hundred real homework questions; math answers can usually be checked for correctness directly, which is cheaper and more reliable than a judge's score.
For the company: one model for everything is the wrong framing. The data suggests the answer differs by product, and neither decision should rest on a 0.04 difference in an average.
Step 7
How your judgment did
Finish the steps above to fill this in.
What to take away
Test both versions on the same items and work with the differences. Report a range along with the average. Count wins and losses, because the average can hide who wins more often. Remember that ties carry no information and that slicing shrinks your sample. And before paying for a difference, ask whether it's bigger than the noise.
Data: MT-Bench single-answer GPT-4 grades, published by LMSYS with FastChat. Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks.