When a Model Grades Your Model
For subjective quality — "is this reply helpful and on-brand?" — there's no answer key, so teams use a strong model as the grader ("LLM-as-judge"). It's powerful, ubiquitous, and dangerous if no one checks the grader.
An LLM judge can score thousands of open-ended outputs cheaply, which is why your team will use it. But a judge is a model with predictable biases: it tends to prefer the answer shown first, the longer answer, and answers in its own style. Worse, an unvalidated judge gives you a precise-looking number that may be wrong. The non-negotiable: the team must check the judge against humans on the cases that actually matter, and re-check it whenever the rubric or model changes.
"How do you know the judge is right?" A strong team answers: we validated it against human labels on the hard cases, we mitigate position and length bias, and we re-validate on every change. A weak team answers: "it's a really good model." If you hear the second answer, the number on the dashboard is decoration.
If a team says it uses G-Eval, ask what they mean in practice: a rubric-driven LLM judge that works through defined criteria before returning a score. The name is useful interview vocabulary; the governance question is more important: where are the human labels that show this judge follows our rubric on the cases that create risk? G-Eval does not remove the need for calibration, slices, or re-validation after a model or rubric change.
The dashboard says 8.4
Your quality dashboard, powered by an LLM judge, shows "8.4/10" and has for months. Leadership loves it. What's the one question that determines whether that 8.4 means anything?
"How well does that judge agree with humans — on the hard cases?" If no one has checked, 8.4 is a number a model made up; it could be systematically lenient, or biased toward long answers, and you'd never know. A trustworthy 8.4 comes with "we sampled 100 cases, had humans grade them, and the judge agrees 88% on the ambiguous ones." Also be wary of a fuzzy 1–10 scale — judges cluster everything at 7–8, so the "8.4" may carry far less information than it appears to. Push the team toward validated, low-cardinality judgments.
PatchProof: the "4.1 and climbing" that wasn't
PatchProof grades its own AI code-reviewer with a second AI acting as judge, scoring each review 1–5. The dashboard looked great: a 4.1 average, rising after a model upgrade — a clear signal to ship. Validation told a different story. Checked against human labels, the judge agreed 91% overall but only 58% on security-relevant changes — weakest exactly where a miss ships a vulnerability. Worse, swapping the order of the two options being compared flipped 20% of verdicts, and the "improved" model had simply written longer reviews, which the judge rewarded regardless of correctness.
A fifth of the "improvement" was layout and wordiness, not quality. After fixing the rubric and always comparing in both orders, the true picture emerged and the launch was gated. The leadership lesson: an AI judge is a measuring instrument, and an unvalidated one can quietly flatter you — always ask "how do we know the judge is right, especially on the cases that matter?"
This chapter: Remi's helpfulness dashboard reads 4.2, up from 3.8 after a prompt tweak — looks like a win. But validated against human labels it's 88% overall and only 56% on the hard refund slice, and the gain is mostly padding: wordier replies score higher on identical facts. Meridian holds Remi's launch rather than trust a flattering judge. (Ch 8: they re-check Remi once it starts issuing refunds.)
Quiz · Chapter 5
- An LLM-as-judge is:
- The key question about any judge-based score is:
- A known bias of LLM judges is:
- "We trust the judge because it's a top model" is: