LLM-as-Judge

When there's no answer key and "good" is a matter of judgment, you ask a capable model to do the grading. It's the technique an evaluation engineer uses most, and without checks it will hand you confident numbers that are wrong.

Here's the problem in miniature. A support bot replies, "I've refunded your order; you'll see it in 3 to 5 business days." Is that good? There's nothing to diff it against. It depends on whether it's accurate, on-brand, complete and safe, and you have ten thousand replies a day. So you give each reply and a rubric to a strong model and ask it to grade. The rest of this chapter is about making that grade something you can trust.

Pointwise and pairwise

A pointwise judge takes one answer and a rubric and returns a score or label. That's useful for tracking a metric over time. A pairwise judge takes two answers and picks the better one. For "did my change help?" pairwise usually works better, because models, like people, compare more reliably than they score in the abstract. Suppose quality really did improve a little between two prompts. A 1-to-5 judge may give both a 4, and you'd conclude nothing changed. Head to head, the same judge picks the new version 7 times out of 10.

Known biases, with numbers

Position bias. Judges tend to prefer whichever answer they read first. One team found the judge chose the first answer on 9 of 50 close pairs, an 18% bias rate. The fix is to run both orders and only keep verdicts that agree.

Verbosity bias. A short, correct reply scored 3.6 out of 5, and a padded reply with the same facts scored 4.3. If your rubric doesn't say "don't reward length," you'll slowly optimize toward waffle.

Self-preference. Judges tend to like outputs in their own model family's style. If the judge and the system are the same model, be skeptical of "the new version is better" until you've checked.

Score compression. On a 1-to-10 scale, almost everything lands between 6 and 8. That's the best argument for small label sets like yes/no or pass/fail.

Lab code · runs free (mock judge by default)

USE_REAL_API = False        # free mock by default; True needs a key

if USE_REAL_API:
    import anthropic
    from pydantic import BaseModel
    client = anthropic.Anthropic()
    class Verdict(BaseModel):
        reason: str        # reasoning BEFORE the label
        winner: str        # "A" or "B"
    def compare(question, a, b):
        r = client.messages.parse(
            model="claude-opus-4-8", max_tokens=400,
            messages=[{"role": "user", "content":
                f"Pick the more accurate, NOT-padded reply. Reason, then winner."
                f"\n\nQ:{question}\n\nA:\n{a}\n\nB:\n{b}"}],
            output_format=Verdict)
        return r.parsed_output.winner
else:
    # Mock with a built-in tilt: it leans toward slot A on close calls
    def compare(question, a, b):
        return "A" if len(a) >= len(b) else "B"   # length/position-ish bias

def robust_compare(question, x, y):
    """Run both orderings; trust only when they agree."""
    x_as_A = (compare(question, x, y) == "A")   # x in slot A
    x_as_B = (compare(question, y, x) == "B")   # x in slot B
    if x_as_A == x_as_B:
        return "x" if x_as_A else "y"
    return "biased"                             # flipped with position

PAIRS = [   # (question, reply_x, reply_y) — close in quality
    ("refund?", "Refunded; 3-5 days.", "Refunded today, arrives in 3-5 days."),
    ("hours?",  "9 to 5, Mon-Fri.",    "We're open 9-5 Monday through Friday."),
    ("reset?",  "Use 'Forgot password'.", "Click the 'Forgot password' link."),
]
flips = sum(robust_compare(q, x, y) == "biased" for q, x, y in PAIRS)
print(f"position-biased on {flips}/{len(PAIRS)} pairs ({flips/len(PAIRS):.0%})")

    def compare(question, a, b):
        return "A"        # always pick the first slot — pure position bias