Free lesson · The AI Evaluation Engineer · Lesson 5
Do Your Human Raters Agree?
Your company is about to pay people to rate chatbot responses, and those ratings will be the "ground truth" everything else is measured against, including the AI judges from the last lesson. Before spending the budget, you need to know how much a single rating can be trusted, how many raters each item needs, and whether the rating scale itself is a good one.
How this lesson works
You'll rate five real responses yourself, compute a chance-corrected agreement score by hand, then measure agreement across 23,652 responses and plan a real labeling budget. Along the way you'll see a common mistake that makes agreement look worse than it is. Plan on about an hour, with a calculator for Step 2.
The data
HelpSteer2 is a dataset NVIDIA published for training and testing AI models. Contractors hired through Scale AI, trained with written guidelines and example ratings, rated chatbot responses from 0 to 4 on five things: helpfulness, correctness, coherence, complexity and verbosity. Every response was rated by at least three people. NVIDIA later released every individual rating as well as the averages, which is what makes this lesson possible.
Source: Wang et al., 2024, "HelpSteer2: Open-source dataset for training top-performing reward models." Dataset: nvidia/HelpSteer2 (the "disagreements" files) on Hugging Face, CC-BY-4.0. Prompts and responses are shown exactly as in the dataset.
Step 1
Rate five responses yourself
For each response, give a helpfulness rating from 0 to 4, where 4 is a response that fully helps with what the person asked and 0 is one that doesn't help at all. Then see what the trained raters said.
First, one prediction
Two trained raters score the same response on this 0 to 4 scale. How often do they give exactly the same number? %
Stop and think
Look at why raters split. Sometimes the response has an error one rater caught and another didn't: the video one claims it can watch a link, which a text model can't; the birthday one depends on who is writing to whom. Sometimes "helpful" itself is unclear, as in the Alzheimer's one, where the response is accurate and well written but never says what causes the disease. Those are different problems with different fixes: better rater training and checking for the first, a clearer rubric for the second.
Step 2
Agreement beyond luck
Raw agreement is misleading when most ratings are the same number. If 90% of all ratings were 4, two raters who ignored the response and just said "4" would agree most of the time. So we compare raw agreement with the agreement you'd get by chance, from how often each score is used.
Chance agreement: square the share of each score and add them up. That's the chance two raters picking at random, with these habits, pick the same score.
Kappa = (raw agreement − chance) ÷ (1 − chance). 0 means no better than luck; 1 means perfect agreement.
Here are the real numbers for two of the five attributes, using two raters per response:
Attribute
Raw agreement
Share of 0s
1s
2s
3s
4s
Helpfulness
43%
0.10
0.09
0.16
0.23
0.42
Coherence
58%
0.01
0.03
0.07
0.20
0.69
Your turn · calculator
Helpfulness: chance agreement kappa
Coherence: chance agreement kappa
Which do raters agree on more, once luck is removed?
Coherence has the higher raw agreement, 58% against 43%, and the lower kappa. Raters agree on coherence mostly because almost everything gets a 4: 69% of all coherence ratings. With habits like that, two raters would agree 52% of the time by luck alone. Take that away and they barely agree on which responses are less coherent, which is the only part of the rating that tells you anything.
Both kappas are low. On the most widely used scale (Landis and Koch), 0.21 to 0.40 counts as "fair" agreement and anything below 0.20 as "slight." Keep that in mind the next time someone reports "our labels agree 58% of the time" as a sign of quality.
Read agreement() first: it's your Step 2 calculation, plus a weighted kappa that gives partial credit when raters are one point apart instead of treating a 3 against a 4 the same as a 0 against a 4. You should see:
Helpfulness kappa is 0.22 but weighted kappa is 0.51. What does that gap tell you about how raters disagree?
Turning helpfulness into yes/no raises kappa from 0.22 to 0.41. Should you switch your rubric to yes/no? What do you lose?
1. Most disagreement is small. Raters rarely match exactly, but they're usually close: 78% are within one point, and giving credit for near misses more than doubles the agreement score. The scale has more precision than the raters. A "3" from one person and a "4" from another often mean the same judgment.
2. It depends on what you'll use the rating for. If the decision is "good enough to ship or not," a yes/no label measures that decision directly and people agree on it more. If you need to rank responses or train a model to tell good from better, yes/no throws away information you need. A common compromise is to collect the 0 to 4 rating, report weighted agreement, and make decisions on a threshold like "3 or higher." Whatever you choose, report agreement for the version of the label you'll use.
Step 4
A common way to get this wrong
Every response in this dataset was rated by at least three people. When those raters disagreed by more than 2 points on helpfulness, up to two more were added. So some responses have 3 ratings and some have 5. Now run:
python raters_lesson.py pooling
Helpfulness agreement computed three ways
one pair per response pairs 23,652 exact 43% kappa 0.22 weighted 0.51
every pair pooled pairs 106,386 exact 39% kappa 0.18 weighted 0.40
By how many raters a response got
3 raters 16,390 responses exact 48% kappa 0.27 correlation 0.62
5 or more raters 4,786 responses exact 31% kappa 0.09 correlation 0.23
Explain it
Pooling every pair of raters seems like it uses more data, so it should be better. Why does it give lower agreement, and which number should you report?
A response with 5 raters produces 10 pairs; one with 3 raters produces 3. And the responses with 5 raters are exactly the ones picked for extra raters because people disagreed on them. Pooling every pair counts the most contested responses more than three times as heavily as the rest, so the number mostly describes the hard cases. Taking one pair per response counts each response once.
The second table shows how different the two groups are: correlation 0.62 where the first raters agreed well enough, 0.23 on the responses that needed more raters. When you report agreement, say how pairs were counted, and watch for anything in the collection process, like extra raters on disagreements, that makes some items different from others.
Disagreement isn't always a matter of taste, either. Here's a response from the dataset that five people rated:
The response says a cubic foot holds 27.8 US gallons; the real figure is about 7.5. Four raters gave it 0 and one gave it 4. One rater missed a factual error. Regular checks, such as mixing in items with known correct ratings and reviewing raters whose scores drift from everyone else's, catch this.
Step 5
How many raters per item?
Averaging several raters cancels out some of their individual noise. How much? Run:
python raters_lesson.py raters
Correlation between two single raters on helpfulness: 0.51
Reliability of the AVERAGE of k raters (Spearman-Brown: k*r / (1 + (k-1)*r)):
1 raters 0.51
2 raters 0.67
3 raters 0.76
5 raters 0.84
10 raters 0.91
Reliability here means how well one group of raters' averages would match another group's. Going from 1 to 3 raters helps a lot; going from 5 to 10 helps much less.
Decide
Your budget covers 3,000 ratings. Option A: 3,000 responses, one rater each. Option B: 1,000 responses, three raters each. Which would you choose for each purpose?
Building a small set of trusted labels to check an AI judge against:
Comparing the average helpfulness of two chatbot versions:
Trusted labels: B. Each label will be used on its own, as the answer the judge is checked against. A single rater's label has a reliability of 0.51; the average of three has 0.76. With one rater, a large part of the judge's "errors" would be rater errors, like the gallons rating.
Comparing two chatbots: usually A. Here you only care about the average over many responses, and random rater noise averages out across responses just as it does across raters. What doesn't average out is which questions happened to be in the test, the lesson of the first lesson. More responses shrink that. The exception: if raters are biased in a consistent direction, averaging won't remove it, so spot-check with a second rater on a sample either way.
Step 6
Design a labeling plan, no hints
A company's support assistant answers billing questions. Leadership wants a weekly number: "what share of answers were correct?" You have budget for 1,500 ratings a week from trained contractors.
Your plan
Decide the rating scale, the number of raters per answer, how you'll measure and report agreement, and what happens when raters disagree. Use what you found in this lesson.
Scale: yes/no for "correct," because that's the number leadership wants and people agree on binary decisions more (kappa 0.41 against 0.22 here). Add a short list of error types (wrong amount, wrong policy, missing step) so a "no" says why. Write examples of each into the guidelines.
Raters: for the weekly number, one rater on each of about 1,200 answers, with a second rater on 300 of them. The weekly number is an average, so more answers matter more than more raters per answer, and the double-rated 300 let you measure agreement every week.
Agreement: report kappa on the double-rated sample, counting each answer once, next to raw agreement and how often "correct" is used. If most answers are correct, raw agreement will look high even if raters agree on little else, the coherence trap from Step 2. Watch the trend: a falling kappa means the guidelines, the raters or the kinds of questions have changed.
Disagreements: send them to a senior reviewer who decides and notes why. Turn repeated reasons into guideline updates, and add known-answer items (like the gallons one) to catch raters who miss factual errors. Keep the disagreements instead of dropping or averaging them: they show where your rubric is unclear and where your assistant's hardest cases are.
Step 7
How your judgment did
What to take away
Raw agreement can look good when most ratings are the same score, so always compare with chance. Report how you counted pairs, and watch for items chosen because raters disagreed. Match the label to the decision it serves. Use several raters when each label must be trusted on its own, and more items when you only need an average. And treat disagreements as information: some are unclear rubrics, some are rater mistakes.
Data: HelpSteer2 individual annotations, NVIDIA, CC-BY-4.0. Wang, Z. et al. (2024). HelpSteer2: Open-source dataset for training top-performing reward models. Kappa: Cohen, J. (1960). Spearman-Brown prediction formula: Spearman (1910), Brown (1910).