The Four Parts of Every Eval

Once you accept that quality is a measurement, you need to decide what to measure and when. This chapter gives you a small mental model that every eval tool and framework follows, so you can recognize the same pieces wherever you work.

Offline and online

Offline evals run before you deploy, on a fixed set of cases, usually in CI. They're fast and repeatable, and they answer "should we ship this change?" Online evals run on live traffic and answer "is it working for real users right now?" Offline covers the cases you thought of in advance. Online is how you find the ones you didn't.

With a reference or without

Some outputs have a known right answer you can compare against, like a refund amount or a date. That's reference-based scoring. Many don't: there's no single correct reply to "can you explain this charge?" For those you judge the output on its own qualities, such as whether it's supported by the source, correctly formatted or polite. That's reference-free scoring, and later chapters cover how to do it well.

The four parts

Any eval you'll meet has the same four pieces. There's a dataset of cases. There's a task, the harness that runs your system on each case. There's a scorer that turns each output into a number or label. And there's a report that rolls those scores up into something you can decide with, broken down by slice and tracked over time. If you can name the four for a system, you understand how its quality is measured. If one is missing, you've found the gap.

Lab code (key parts) · runs free

POLICY = """Refund policy
1. Duplicate charge: refund the duplicate charge in full.
2. Monthly plan cancelled: refund the unused days, prorated: price x (30 - days used) / 30.
3. Annual plan cancelled within 14 days of purchase: full refund.
4. Annual plan cancelled after 14 days: prorated: price x (365 - days used) / 365.
Round refunds to the cent."""

def correct_answer(t):
    """The answer the POLICY requires. This is the ground truth."""
    if t["kind"] == "balance":
        return f"${t['balance']:.2f}"
    if t["kind"] == "price":
        return f"${PRICES[t['plan']]:.2f}"
    price, used = PRICES[t["plan"]], t["days_used"]
    if t["kind"] == "duplicate":
        return f"${price:.2f}"
    if t["kind"] == "cancel_monthly":
        return f"${price * (30 - used) / 30:.2f}"
    if used <= 14:
        return f"${price:.2f}"
    return f"${price * (365 - used) / 365:.2f}"

def bot_reply(t):
    if USE_REAL_API:
        return real_model_reply(t)
    # A stand-in for a model that read the policy. Read it closely.
    if t["kind"] == "balance":
        return f"Your balance is ${t['balance']:.2f}."
    if t["kind"] == "price":
        return f"The {t['plan']} plan is ${PRICES[t['plan']]:.2f}."
    price, used = PRICES[t["plan"]], t["days_used"]
    if t["kind"] == "duplicate":
        return f"Sorry about that. I've refunded ${price:.2f}."
    if used <= 14:                                   # the 14-day rule...
        return f"You're within 14 days, so I've refunded the full ${price:.2f}."
    days_in_period = 30 if t["plan"] == "monthly" else 365
    return f"I've refunded ${price * (days_in_period - used) / days_in_period:.2f} for the unused time."

def score(t, reply):
    want = float(correct_answer(t)[1:])
    got = first_amount(reply)
    ok = got is not None and abs(got - want) < 0.01
    dollars_wrong = 0.0 if ok or t["intent"] != "refund" else (got or 0.0) - want
    return ok, dollars_wrong

overall accuracy   0.96  (192 of 200)
  account          1.00  (89 of 89)
  plan             1.00  (69 of 69)
  refund           0.81  (34 of 42)
    duplicate      1.00  (14 of 14)
    cancel_monthly 0.50  (8 of 16)
    cancel_annual  1.00  (12 of 12)

money: over-refunded $59.93, under-refunded $0.00 on 200 tickets
       about $2,996 refunded wrongly per 10,000 tickets

failures grouped by pattern:
    8 x cancel_monthly, used <= 14 days
        e.g. "I'm cancelling my monthly plan after 5 days. What's my refund?"
        bot said: You're within 14 days, so I've refunded the full $29.00.
        policy says: $24.17

verdict: HOLD (limit $1,000 wrong per 10,000 tickets)