LLM-as-Judge

When there's no answer key and "good" is a matter of judgment, you make a capable model do the grading. It's the single most-used technique in the modern eval engineer's day — and the one most likely to quietly lie to you if you don't pin it down.

A support bot replies: "I've refunded your order — 3–5 business days." Good answer? No reference to diff against; "good" depends on accuracy, brand, completeness, safety. You have ten thousand a day. So you hand the answer plus a rubric to a strong model and ask it to grade. The rest of this chapter makes that judgment trustworthy — an unvalidated judge is just an opinion wearing a number.

Pointwise vs. pairwise

Pointwise: one answer + rubric → a score or label. Good for tracking a metric over time. Pairwise: two answers → which is better. The workhorse for "did my change help?", because judges (like people) compare far more reliably than they score absolutely. Worked: if real quality rose a hair between two prompts, a pointwise 1–5 judge scores both "4" (you conclude "no change"); head-to-head it picks the new one 7 times out of 10. Same judge, same answers — the comparative framing recovered a signal the absolute scale flattened to zero.

The biases, in real numbers

• Position bias — prefers whichever answer is shown first. One team saw the judge pick "shown first" on 9 of 50 close pairs: an 18% bias rate. Fix: run both orderings, keep the verdict only when they agree.
• Verbosity bias — a crisp correct reply scores 3.6/5; a padded version of the same facts scores 4.3/5. If your rubric doesn't say "don't reward length," you're optimizing toward waffle.
• Self-preference — favors its own family's style. If judge and system are the same model, distrust "the new one is better" until you validate.
• Leniency / compression — on a 1–10 scale everything lands 6–8. The best argument for low-cardinality outputs.

Lab code — runs free (mock judge by default)

USE_REAL_API = False        # free mock by default; True needs a key

if USE_REAL_API:
    import anthropic
    from pydantic import BaseModel
    client = anthropic.Anthropic()
    class Verdict(BaseModel):
        reason: str        # reasoning BEFORE the label
        winner: str        # "A" or "B"
    def compare(question, a, b):
        r = client.messages.parse(
            model="claude-opus-4-8", max_tokens=400,
            messages=[{"role": "user", "content":
                f"Pick the more accurate, NOT-padded reply. Reason, then winner."
                f"\n\nQ:{question}\n\nA:\n{a}\n\nB:\n{b}"}],
            output_format=Verdict)
        return r.parsed_output.winner
else:
    # Mock with a built-in tilt: it leans toward slot A on close calls
    def compare(question, a, b):
        return "A" if len(a) >= len(b) else "B"   # length/position-ish bias

def robust_compare(question, x, y):
    """Run both orderings; trust only when they agree."""
    x_as_A = (compare(question, x, y) == "A")   # x in slot A
    x_as_B = (compare(question, y, x) == "B")   # x in slot B
    if x_as_A == x_as_B:
        return "x" if x_as_A else "y"
    return "biased"                             # flipped with position

PAIRS = [   # (question, reply_x, reply_y) — close in quality
    ("refund?", "Refunded; 3-5 days.", "Refunded today, arrives in 3-5 days."),
    ("hours?",  "9 to 5, Mon-Fri.",    "We're open 9-5 Monday through Friday."),
    ("reset?",  "Use 'Forgot password'.", "Click the 'Forgot password' link."),
]
flips = sum(robust_compare(q, x, y) == "biased" for q, x, y in PAIRS)
print(f"position-biased on {flips}/{len(PAIRS)} pairs ({flips/len(PAIRS):.0%})")

    def compare(question, a, b):
        return "A"        # always pick the first slot — pure position bias