Metrics & Scorers

A scorer turns an output into a number or label. The art is matching the scorer to the task — and knowing each one's blind spots, out loud.

The families of scorers

• Deterministic / rule-based — exact match, regex, JSON-schema validity, "does it cite a source," code that compiles, math that equals the answer. For code and math, execution-based scoring (run it, check the result) is the gold standard. Cheap and perfectly reliable — use wherever the task allows.
• Statistical overlap — BLEU, ROUGE measure n-gram overlap with a reference. Cheap but weak: they reward surface words and punish valid paraphrases. Coarse signals, never quality verdicts.
• Semantic similarity — embed output and reference, take cosine. Captures meaning better than n-grams, but tells you "about the same thing," not "correct."
• Model-graded (LLM-as-judge) — the flexible, dominant approach for open-ended quality. Its own chapter (5).

Many "LLM tasks" are secretly classification

Routing, intent detection, safety filtering, extraction correctness — the old vocabulary applies and interviewers expect it. Precision: of what I flagged, how much was right. Recall: of what I should have flagged, how much I caught. F1: their harmonic balance. Know which to optimize: a safety filter cares about recall (missing unsafe content is the costly error); a "delete this email" action cares about precision.

Lab code — runs free

import json

def exact_match(out, ref):
    return float(out.strip() == ref.strip())

def valid_json(out, _ref):
    try:
        json.loads(out); return 1.0
    except ValueError:
        return 0.0

# Safety filter: 1 = flagged unsafe. Compare predictions to ground truth.
y_true = [1, 1, 1, 0, 0, 0, 0, 0, 1, 1]   # what was actually unsafe
y_pred = [1, 1, 0, 0, 0, 1, 0, 0, 1, 0]   # what the filter flagged

tp = sum(t == 1 and p == 1 for t, p in zip(y_true, y_pred))
fp = sum(t == 0 and p == 1 for t, p in zip(y_true, y_pred))
fn = sum(t == 1 and p == 0 for t, p in zip(y_true, y_pred))

precision = tp / (tp + fp)
recall    = tp / (tp + fn)
f1 = 2 * precision * recall / (precision + recall)
print(f"precision={precision:.2f}  recall={recall:.2f}  f1={f1:.2f}")

precision=0.75  recall=0.60  f1=0.67