Chapter 6

Human Evaluation & Annotation

Automated scorers and model judges all rest on human judgment in the end. For subjective quality, people are still the ground truth, and the evaluation engineer is usually the one who designs, runs and checks the labeling work.

Case study

Ravel's ground truth wasn't

Ravel did the right thing and checked its LLM judge against human labels. The judge kept disagreeing with the people in ways nobody could explain. The trouble was the labels. Three annotators working from a one-line guideline agreed with each other only 64% of the time, so the "truth" was really three different opinions.

Ravel fixed the human side first. They wrote a rubric with examples of each label, ran a calibration round on shared cases, and had a senior reviewer settle disagreements. Agreement rose to 89%, and only then did the judge's numbers start to mean something.

Where to spend people

Human labeling is slow and costly, so use it where nothing else will do: building golden sets, checking judges, making high-stakes or subjective calls, and collecting preference data. Automate the rest and spot-check it.

The skill that decides whether the labels are any good is rubric design. "Rate quality from 1 to 5" gets you noise. A good rubric defines each criterion concretely and breaks the judgment into the smallest decision people can make reliably, which is often a yes or no per dimension ("Is every claim supported by the source?"). The same discipline makes model judges better too.

Key metric: agreement

Inter-annotator agreement tells you whether your labelers agree with each other. Cohen's kappa is the common measure, and it corrects for agreement that would happen by chance. Raw agreement can look high while kappa is only middling. Low kappa usually means the rubric is ambiguous, and a model judge can't be expected to match people who don't agree among themselves.

Preference data feeds training

Human choices of the form "A is better than B" are useful twice. They validate systems, and they're the raw material for post-training methods such as RLHF and DPO. In a senior interview it helps to show you know evaluation data often ends up improving the model itself.

Try it · ~10 minFree · no API key

Compute Cohen's kappa by hand

Two annotators have labeled ten answers as grounded (1) or not (0). Compute raw agreement and kappa using only the standard library, and watch how much the raw number drops once you remove what chance alone would produce.

Lab code · runs free

A = [1, 1, 0, 1, 0, 0, 1, 1, 0, 1]   # annotator A: grounded? 1/0
B = [1, 0, 0, 1, 0, 1, 1, 1, 0, 1]   # annotator B
n = len(A)

po = sum(a == b for a, b in zip(A, B)) / n      # observed (raw) agreement
pa1, pb1 = sum(A) / n, sum(B) / n               # each rater's rate of "1"
pe = pa1 * pb1 + (1 - pa1) * (1 - pb1)           # agreement expected by chance
kappa = (po - pe) / (1 - pe)

print(f"raw agreement = {po:.2f}")
print(f"chance        = {pe:.2f}")
print(f"kappa         = {kappa:.2f}")

Worked solution

raw agreement = 0.80
chance        = 0.52
kappa         = 0.58

Raw agreement of 0.80 makes the rubric look solid. Both annotators say "grounded" most of the time, though, so they'd agree on about 0.52 of cases just by chance. Kappa, which removes that, is 0.58, usually described as moderate. If it fell toward 0.3, you'd rewrite the rubric with sharper criteria and examples for each label before trusting either the people or any judge measured against them.

Running case · Meridian × Remi

Two Meridian support leads label 120 of Remi's refund replies as acceptable or not, working from a written rubric. They start at 0.72 agreement, talk through the disagreements, tighten the guideline and reach 0.9. That labeled refund set is what they used in Chapter 5 to discover the judge was right only 56% of the time on refunds.

If you're starting out

Annotation work sounds unglamorous, but many people's first evaluation tasks involve writing rubrics, running a labeling round or checking agreement. Doing it well is a visible way to earn trust on a new team. If you get the chance, be the person who notices that two labelers disagree and turns that into a clearer guideline.

For interviews, be ready to explain why kappa exists. "Raw agreement counts matches that would happen by luck, especially when one label is much more common, and kappa subtracts those" is enough.

If you run the team

Labeling budgets land on your desk sooner or later. Suppose the team asks to have people label 50,000 outputs and mentions that the two reviewers "often disagree." The disagreement is the important part. Paying for 50,000 labels from an unclear rubric buys 50,000 noisy labels. Fund a rubric fix first, confirm agreement goes up, and then label a focused set: a golden set and a set for checking the judge.

After that, people set the standard and a checked model judge applies it at scale. The cost of human labels includes writing guidelines, calibrating reviewers and settling disagreements, and skipping that work is how teams end up with expensive evals that mislead them.

Back Continue