Human Evaluation & Annotation

Automated scorers and judges are scaffolding on top of human judgment. Humans remain the ground truth for subjective quality — and the eval engineer usually designs, runs, and quality-controls that human process.

When to spend humans, and the skill that decides it

Humans are slow and expensive — spend them where they're irreplaceable: creating golden sets, validating judges, high-stakes or subjective calls, and preference data. Everything else is automated and spot-checked. The real skill is rubric design: "rate quality 1–5" yields noise; a good rubric defines each criterion concretely and reduces judgment to the smallest reliable decision — often a binary per dimension (grounded? yes/no). The same rubric discipline works for humans and LLM judges.

Preference data closes the loop to training

Pairwise human preferences ("A is better than B") do double duty: they validate systems and they're the raw material for post-training (RLHF/DPO). Knowing that evaluation quietly becomes the data engine for model improvement signals seniority.

Lab code — runs free

A = [1, 1, 0, 1, 0, 0, 1, 1, 0, 1]   # annotator A: grounded? 1/0
B = [1, 0, 0, 1, 0, 1, 1, 1, 0, 1]   # annotator B
n = len(A)

po = sum(a == b for a, b in zip(A, B)) / n      # observed (raw) agreement
pa1, pb1 = sum(A) / n, sum(B) / n               # each rater's rate of "1"
pe = pa1 * pb1 + (1 - pa1) * (1 - pb1)           # agreement expected by chance
kappa = (po - pe) / (1 - pe)

print(f"raw agreement = {po:.2f}")
print(f"chance        = {pe:.2f}")
print(f"kappa         = {kappa:.2f}")

raw agreement = 0.80
chance        = 0.52
kappa         = 0.58