When there is no answer key and "good" is a matter of judgment, you make a capable model do the grading. This is the single most-used technique in the modern eval engineer's day — and the one most likely to quietly lie to you if you don't pin it down.
Here is the situation you cannot escape. A customer-support bot replies: "I've refunded your order — you'll see it in 3–5 business days." Is that a good answer? There is no reference string to diff against. "Good" depends on whether it's accurate, on-brand, complete, and safe. You have ten thousand of these a day. You cannot read them all, and assert is useless. So you hand the answer, plus a rubric, to a strong model and ask it to grade. That's LLM-as-judge. The rest of this chapter is about making that judgment trustworthy — because an unvalidated judge is just an opinion wearing a number.
There are two ways to ask, and the choice changes your results more than people expect.
Pointwise (absolute): show the judge one answer and a rubric; get back a score or label — "rate faithfulness 1–5" or "grounded: yes/no." Simple, scales, good for tracking a metric over time.
Pairwise (comparative): show the judge two answers — say, your old prompt vs. your new prompt — and ask which is better. This is the workhorse for the question you actually have most days: "did my change help?"
Why does pairwise win for that question? Because judges (like people) are far more reliable at comparing than at assigning absolute scores. Watch it happen with numbers. Suppose your real, ground-truth quality went up by a hair between two prompt versions. A pointwise judge on a 1–5 scale will often score both versions "4" — it can't resolve the difference, and you conclude "no change." Ask it head-to-head and it picks the new one 7 times out of 10. Same judge, same answers; the comparative framing exposed a signal the absolute framing flattened to zero.
Use pointwise to monitor a quality metric over time (a dashboard line). Use pairwise to decide whether a specific change shipped an improvement (a release gate). Most teams need both, for different jobs.
A judge is a model, and models have systematic tilts. If you only memorize their names you'll nod in an interview and still ship a broken judge. So here is each one as it actually shows up.
Show the judge answer X in slot A and answer Y in slot B; it prefers X. Swap them — X now in slot B — and it still leans toward whatever is in the first slot. Concretely: across 50 head-to-head pairs where the two answers were genuinely close, one team saw the judge pick "whichever was shown first" on 9 of them. That's an 18% position-bias rate — nearly one in five verdicts driven by layout, not quality. You'll build the fix yourself in the lab below: run both orderings, keep the verdict only when they agree.
Judges tend to reward longer, more elaborate answers even when the short one is just as correct. Worked example: a crisp, correct two-sentence reply scores 3.6/5; a padded, hedge-filled version of the same facts scores 4.3/5. Nothing improved except word count. If your rubric doesn't explicitly say "do not reward length," you are quietly optimizing your product toward waffle.
A judge often favors answers written in its own model family's style, or that it would have written itself. If the system you're grading and the judge are the same model, treat every "the new version is better" with suspicion — and validate against humans (next section) before you believe it.
On a 1–10 pointwise scale, judges cluster: almost everything lands 6–8. Your "scores" then carry maybe two bits of real information dressed up as ten. This is the single best argument for low-cardinality outputs — a binary or a 1–3 the judge can actually use consistently.
This is the section that separates someone who uses LLM-as-judge from someone who can be trusted to. If a model grades your model, you must measure how good the grader is — against the only ground truth that counts, humans.
The method: take a small set (50–200 cases), have a human label each one, then run your judge on the same set and measure agreement. But the headline number hides a trap. Suppose your judge agrees with humans 95% on the easy cases (obviously-good and obviously-bad answers) and 55% on the hard, ambiguous cases — and the hard cases are exactly the ones where decisions actually get made. If your test set is 80% easy, the blended figure reads a reassuring "87% agreement," and it's useless, because the judge is a coin flip precisely where you need it. Always report judge–human agreement on the decision-relevant slice, not the blended average. (That's the same slicing instinct from the datasets chapter — it shows up everywhere.)
If an interviewer asks "how do you trust an LLM judge?", the senior answer is three beats: (1) validate against human labels on the hard slice, (2) mitigate the known biases (position-swap, length-neutral rubric, low-cardinality output), (3) re-validate whenever the rubric or the underlying model changes. Say that and you sound like you've shipped one.
Here is the same judge, written badly and then well:
Before — vague, fragile
prompt = f"Rate the quality of this answer from 1 to 10.\n\n{answer}"
# Returns "8" almost every time. No rubric, no reasoning, scale it can't use.
# You cannot tell what "8" means or defend it when a human disagrees.After — specific, auditable, low-cardinality
RUBRIC = """Grade whether the ANSWER is fully supported by the CONTEXT.
- grounded = true : every claim in the answer is stated in the context
- grounded = false : the answer adds, changes, or invents any fact
Do NOT reward length or fluency. A short answer that is fully supported
beats a long one that is not. Explain your reasoning in one sentence,
then give the verdict."""Notice the after-version does three jobs at once: it defines the criterion, it explicitly neutralizes verbosity bias, and it forces a binary the judge can hit consistently. You now know exactly what the number means and can hand-check it against a human.
You'll write a real LLM-as-judge with the Anthropic SDK, run it head-to-head on two answers, and then measure its own position bias by swapping the order. This is the exact loop you'd run on the job.
Setup: pip install anthropic pydantic, then export ANTHROPIC_API_KEY=....
Step 1. Write a pairwise judge that returns a structured verdict (reason first, then winner).
Step 2. Wrap it so it runs both orderings and only trusts a verdict when the two agree.
Step 3. Run it on the 5 pairs below and report: how many flipped with order? That flip rate is your judge's position-bias rate.
Your goal: a number — "the judge flipped on N/5 pairs" — and a one-sentence read on whether you'd trust its raw verdicts.
Starter code
import anthropic
from pydantic import BaseModel
client = anthropic.Anthropic()
class Verdict(BaseModel):
reason: str # force the reasoning BEFORE the label
winner: str # "A" or "B"
RUBRIC = ("Two customer-support replies answer the same question. "
"Pick the reply that is more accurate and helpful, and is NOT "
"padded with unnecessary words. Give a one-sentence reason, "
"then the winner.")
def compare(question, reply_a, reply_b) -> str:
r = client.messages.parse(
model="claude-opus-4-8",
max_tokens=400,
messages=[{"role": "user", "content": (
f"{RUBRIC}\n\nQUESTION:\n{question}\n\n"
f"REPLY A:\n{reply_a}\n\nREPLY B:\n{reply_b}"
)}],
output_format=Verdict,
)
return r.parsed_output.winner # "A" or "B"
# TODO (Step 2): run BOTH orderings; trust only when they agree.
# TODO (Step 3): loop over PAIRS, count flips, print the bias rate.
PAIRS = [
# (question, concise_correct_reply, padded_reply)
("Where is my refund?",
"Refunded today; 3-5 business days to appear.",
"Thank you so much for reaching out! I completely understand how "
"important this is. I'm pleased to let you know your refund has been "
"processed and should arrive within 3-5 business days."),
# ...add 4 more of your own...
]Step 2 is the whole point of the lab — the swap-and-agree wrapper:
def robust_compare(question, x, y) -> str:
"""Run both orderings. x is the SAME answer in both calls; only its
slot changes. If the winner tracks the answer, trust it. If it tracks
the slot, the judge is position-biased on this pair."""
first = compare(question, x, y) # x sits in slot A
second = compare(question, y, x) # x sits in slot B
x_won_as_a = (first == "A")
x_won_as_b = (second == "B")
if x_won_as_a == x_won_as_b:
return "x" if x_won_as_a else "y" # consistent verdict
return "biased" # flipped with position
flips = sum(robust_compare(q, a, b) == "biased" for q, a, b in PAIRS)
print(f"position-biased on {flips}/{len(PAIRS)} pairs "
f"({flips/len(PAIRS):.0%})")What you should see. On pairs where the two replies are genuinely close in quality, a meaningful fraction come back "biased" — the judge flipped its winner purely because the answer moved slots. On a real run of ~50 close pairs that lands around 15–20%. The fix is already in your hands: robust_compare only returns a verdict when both orderings agree, which converts most of those silent errors into an honest "biased" you can route to a human.
The lesson, in one line: a single judge call gives you a verdict; two calls give you a verdict and a confidence signal. The second call is the cheapest reliability upgrade in all of eval engineering.
Going further (optional): add a third reply that is correct but very long and see whether your rubric's "not padded" clause actually holds the line, or whether verbosity bias still wins. Then try deleting that clause and watch the scores move — that contrast is the whole verbosity-bias lesson, felt rather than read.
1 — B. Absolute scales flatten small differences; comparative framing recovers the signal. That's exactly why pairwise is the release-gate workhorse.
2 — C. A flip-on-swap is the definition of position bias; the swap-and-agree wrapper is the standard mitigation.
3 — A. The blended average is dominated by easy cases; the judge is a coin flip where it matters. Always report agreement on the decision-relevant slice.
4 — C. Leniency/compression makes fine scales low-information; low-cardinality outputs are more consistent and more defensible.
5 — A. Validation is tied to a specific rubric + model. Change either and you must re-validate — yesterday's calibration doesn't transfer.
If this chapter taught you something, the other eleven will too — each one a deep worked read → a Try-It lab you run → a worked solution → a reasoning quiz. One-time purchase, yours to keep.
Also available: New Grad edition · Manager & Founder edition
Sample chapter · The AI Evaluation Engineer · AI Engineer Dojo · aiengineerdojo.com
If this format lands, the full 12-chapter book gets rebuilt this way: deep worked read → Try-It lab → worked solution → reasoning quiz.