Free lesson · The AI Evaluation Engineer · Lesson 4
Can You Trust an AI Judge?
Your team wants to stop paying people to compare chatbot answers and let GPT-4 do it instead. Your manager asks you one question: can we trust it? By the end of this lesson you'll answer that with evidence, for two different products, and you'll have made and corrected several wrong guesses along the way.
How this lesson works
You'll do the work before you read the explanation. Each step asks you to decide, predict or count something first, and the answer stays hidden until you commit. Being wrong is expected; the wrong guesses are where most of the learning happens. Plan on 60 to 90 minutes, with a pen and paper for Step 2.
The data
Everything here is real. In 2023, researchers at LMSYS asked six chatbots the same 80 questions, in eight categories: writing, roleplay, reasoning, math, coding, extraction, STEM and humanities. Then they showed pairs of answers to people, mostly experts, and asked which was better: answer A, answer B, or a tie. That produced 3,355 human votes. They also had GPT-4 judge the same pairs.
One detail matters a lot later. GPT-4 judged every pair twice, once with the answers in each order. When its two verdicts agreed, that's the verdict. When they disagreed, meaning GPT-4 picked whichever answer happened to be in a particular slot, the dataset marks it "tie (inconsistent)." In this lesson that's called a flip.
Source: Zheng et al., 2023, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." Dataset: lmsys/mt_bench_human_judgments on Hugging Face, CC-BY-4.0.
Step 1
Be the judge
Before you measure a judge, be one. Below are six real pairs from the dataset. For each, read the question and both answers, then make two calls:
Your verdict: is A better, B better, or is it a tie?
Your prediction: did GPT-4 agree with the human raters on this pair?
For the math and logic questions, work out the correct answer yourself first. Don't skim; the point is to feel how hard or easy each call is.
Step 2 opens when you’ve revealed all six pairs.
Stop and think
Before moving on, write down one sentence: what made some pairs easy to judge and others hard? Keep it; you'll need it in Step 5.
Step 2
Count agreement by hand
"Does the judge agree with humans?" sounds like one number. Working it out by hand shows why it isn't.
Below are 12 items from the dataset, picked at random: 6 writing questions and 6 math questions. Each had at least two human raters who chose a winner, and GPT-4 gave a consistent verdict. The table shows every human vote and GPT-4's verdict.
How to count. For human vs human, compare every pair of human votes on the same item. An item with votes A, B, B has three pairs (A-B, A-B, B-B), and one of them agrees. An item with two votes has one pair. For GPT-4 vs human, compare GPT-4's verdict with each human vote separately. An item with votes A, B, B and GPT-4 = A gives 3 comparisons, 1 of which agrees.
Your turn · pen and paper
Count these for the six writing items:
Human pairs that agree: out of pairs
GPT-4 comparisons that agree: out of comparisons
And for the six math items:
Human pairs that agree: out of
GPT-4 comparisons that agree: out of
Writing: humans agree on 4 of 10 pairs (40%). GPT-4 agrees with 6 of 14 human votes (43%). Math: humans agree on 6 of 6 (100%), and GPT-4 agrees with 12 of 12 (100%).
Look at writing first. GPT-4 agrees with people only 43% of the time, which sounds terrible, until you notice that people agree with each other only 40% of the time on the same items. When two careful humans disagree on which opening paragraph is better, there's no single right answer for the judge to match. So "GPT-4 agrees with humans X%" means nothing on its own. You have to compare it with how often humans agree with each other on the same kind of question. That second number is your yardstick: roughly how much agreement is possible on that kind of question at all.
Now math. Everyone agrees, 100%. Hold on to that; in Step 3 the full data will tell a slightly different story, and you'll need to explain why.
Step 3
The full data
Twelve items can mislead you. Now you'll run the numbers on all of them. First, commit to predictions:
Predict first
Across all categories, GPT-4 agrees with human votes about % of the time.
Humans agree with each other about % of the time.
The category where GPT-4 agrees least with humans:
Now run the script. Download judge_lesson.py, put it in an empty folder, and in a terminal run:
pip install pandas pyarrow
python judge_lesson.py
The first run downloads the two data files (about 1.4 MB) from Hugging Face.
Read judge_vs_human() and human_vs_human() in the script before you look at the output. They do exactly what you did by hand in Step 2, on every item. The numbers in brackets are 95% intervals, the range the true value probably sits in given how many votes there are. You should see:
category GPT-4 vs human human vs human GPT-4 flips
writing 73% (65% to 79%, n=142) 68% (58% to 77%, n=92) 9%
roleplay 81% (75% to 86%, n=179) 73% (66% to 80%, n=146) 14%
reasoning 95% (88% to 98%, n=82) 92% (82% to 96%, n=61) 23%
math 94% (87% to 97%, n=94) 96% (89% to 98%, n=90) 19%
coding 86% (79% to 91%, n=116) 92% (84% to 96%, n=77) 20%
extraction 92% (86% to 96%, n=140) 86% (76% to 92%, n=76) 15%
stem 88% (82% to 92%, n=185) 78% (70% to 85%, n=116) 14%
humanities 84% (78% to 89%, n=180) 84% (77% to 90%, n=121) 14%
ALL 86% (83% to 88%, n=1118) 82% (79% to 85%, n=779) 16%
Explain it
Answer these in your own words before opening the discussion:
Your math sample in Step 2 said 100% for both. The full data says 94% and 96%. Give two reasons your sample overstated it.
In writing, roleplay, extraction and STEM, GPT-4 agrees with people more than people agree with each other. Does that mean GPT-4 is a better rater than a person? Be careful: think about what each number compares.
Based on this table alone, would you tell your manager the judge can be trusted?
1. Why the sample said 100%. Six items is very few; with 12 comparisons, a 95% interval around 12 of 12 runs from about 76% to 100%, so "100%" was never a precise number. The sample was also skewed: it only included items where at least two raters picked a winner and GPT-4 gave a consistent verdict. That filter favors clear-cut questions. Small samples and hidden filters are the two most common ways a hand check misleads you.
2. Is GPT-4 a better rater? Not necessarily. "Human vs human" compares two noisy raters with each other; "GPT-4 vs human" compares GPT-4 with one noisy rater. A rater that tends to pick the majority view will agree with a random person more often than two random people agree with each other, without being better at the task. What you can say is that on these categories GPT-4 is about as consistent with people as people are with each other.
3. Can it be trusted? On this table, it looks like yes: 86% overall, above the human ceiling in most categories. Hold that conclusion loosely. The last column, and one line at the bottom of your terminal output, haven't been explained yet.
Step 4
What the agreement number leaves out
Every agreement number in Step 3 counted only decisive verdicts. Pairs where GPT-4 flipped, choosing one answer in one order and the other answer in the reverse order, were left out, along with ties. That's the "GPT-4 flips" column: 16% of all pairs.
Predict first
If flips and ties are counted as a third possible verdict, "tie", instead of being left out, overall agreement drops from 86% to about %.
Flips are most common in:
The last line of your terminal output answers the first question: 67%. The research paper this data comes from reports 66% for the same comparison. Same data, same judge, and the headline moves from 86% to 67% depending on one decision about what to do with flips and ties. When someone quotes you a judge-agreement number, ask how they handled them.
For the second: flips are highest in reasoning (23%), coding (20%) and math (19%), and lowest in writing (9%).
Stop on that for a moment. Math, reasoning and coding are the categories where there is a right answer. On roughly one pair in five, GPT-4's verdict depended on which answer it read first. Writing, where people disagree the most, is where GPT-4 is steadiest. So the headline number was flattering in exactly the places where you'd expect a judge to be most reliable.
Your turn · read the disagreements
Numbers only tell you where to look. Now read what went wrong. Run:
python judge_lesson.py read math
It prints math pairs where every human rater chose one answer and GPT-4 chose the other. Take the first one, question 111, turn 2: the area of the circle through the points (0, 0), (-1, 1) and (3, 3).
Work it out yourself before reading either answer closely. Hint: find a point that is the same distance from all three vertices. Then decide who was right, the human or GPT-4.
Area of the circle:
The center is (1, 2): it's √5 from (0, 0), from (−1, 1) and from (3, 3). So the radius is √5 and the area is 5π, about 15.71. Answer B got that right. Answer A used a correct formula but made an arithmetic slip, getting a radius of 2√45 and an area of 180π. GPT-4 preferred A, the longer, more formula-heavy answer with the wrong result, and the human chose B.
Now compare that with the race question in Step 1, where the human rater picked the wrong answer ("first place") and GPT-4 picked the right one. Both kinds of mistakes are in this data. A disagreement tells you something is off, but not which side is wrong. The only way to know is to check the answer yourself, and on tasks with a right answer, you often can.
Step 5
Make the call
Here's the real decision. Two teams at your company want to replace human raters with GPT-4 pairwise judging.
Team Tutor builds a math homework helper. They compare two versions of their bot on 300 math questions each week and ship the winner.
Team Copy builds a tool that writes marketing emails. They compare two versions on 300 writing prompts each week.
Your recommendation · no hints
For each team, write what you'd tell them: can they use GPT-4 as the judge, and if so, how? Use the numbers from Steps 3 and 4 and at least one thing you saw in Step 1. Be specific enough that the team could act on it.
Team Tutor (math). GPT-4 agrees with people 94% of the time on decisive math verdicts, close to the 96% human ceiling, but it flips on 19% of math pairs, and Step 4 showed it can prefer a confident wrong answer. A reasonable recommendation: use GPT-4, but always judge each pair in both orders and treat a flip as "no verdict," never as a win. Better still, since math has right answers, check the final answer against a reference solution in code and only use the judge for things code can't check, like clarity of explanation. Spot-check a sample of judge decisions by hand every week. With 300 questions and a 19% flip rate, expect around 57 pairs with no verdict; the comparison has to be decided on the rest.
Team Copy (writing). People agree with each other only about 68% of the time on writing, and GPT-4 matches them at 73% while flipping least (9%). GPT-4 is about as consistent as a human rater here, so replacing humans is reasonable. The catch is that "better" is subjective, so the judge's taste becomes the product's taste. Write down what a good marketing email means for this team (clear offer, right length, brand voice) and put it in the judge's rubric, then check each month that the judge's picks match what real customers respond to.
Compare with yours. Did you treat flips as wins or as no verdict? Did you compare with how much humans agree on writing? Did you suggest checking math answers directly instead of trusting a judge? Each of those is a judgment call you'll make in the job.
Step 6
How your predictions did
Your guesses from this lesson, next to what the data showed.
Finish the steps above to fill this in.
What to take away
An agreement number means little until you know three things: how often humans agree on the same task, how flips and ties were counted, and which kinds of questions it covers. And when the judge and the humans disagree, read the pair. Sometimes the judge is wrong, sometimes the human is, and only checking tells you which.
Data: MT-Bench human judgments, LMSYS (lmsys/mt_bench_human_judgments), CC-BY-4.0. Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks. Model answers and votes are shown unedited.