Metrics Without the Math
You'll hear precision, recall, F1, faithfulness. You don't need the formulas — you need to know which one maps to your risk, because the choice is a business decision disguised as a technical one.
The two that matter most for any "flag the bad stuff" system (a safety filter, a fraud check, a moderation step): precision is "of what we flagged, how much was actually bad," and recall is "of what was actually bad, how much did we catch." You usually can't max both — tighten one and the other loosens. The right trade-off depends on which error is more expensive. A safety filter favors recall (missing harmful content is catastrophic; an over-cautious flag is cheap). An auto-delete action favors precision (a wrong deletion is unrecoverable). For open-ended quality, the team will lean on a model-graded score (next chapter).
When your team picks a metric, ask: "which error is more expensive for us — a miss or a false alarm?" That answer should drive whether they optimize recall or precision. If they can't connect the metric to a business cost, the metric is arbitrary.
The 99%-accurate moderation filter
A team reports their content-moderation filter is "99% accurate." Only 2% of content is actually harmful. Why might 99% accuracy be a disaster, and what should you ask for instead?
If 2% of content is harmful, a filter that flags nothing at all is automatically 98% "accurate" — accuracy is meaningless for rare events. The 99% could mean it's catching almost no harmful content. Ask for recall ("of the genuinely harmful items, what fraction did we catch?") and accept the precision trade-off that comes with raising it. For a moderation filter, a recall number you can live with — not a flattering accuracy — is the launch criterion.
SafeGuard: optimizing the wrong number
SafeGuard drove its content filter's precision to 0.95 — of everything it flagged, 95% was truly harmful — and leadership was pleased, until harmful content kept reaching users. The blind spot was recall, sitting at 0.60: the filter missed 40% of genuinely unsafe content. For a safety gate, recall is the number that matters, because a missed harmful item is the costly error and an over-cautious flag is cheap.
They had optimized the metric that looked good instead of the one that matched the cost of their errors. Re-tuned to prioritize recall (with humans reviewing the extra false positives), missed-harmful content dropped sharply. The leadership lesson, in plain terms: which number you chase should be set by which mistake is expensive for you — a filter, a router, and a summarizer each care about different errors, and "accuracy" alone can hide the one that hurts.
This chapter: Meridian's team explains Remi's scorers to leadership without the math. A simple exact check confirms a processed refund matches policy. For routing "possible fraud," they optimize recall — missing a fraud case is the expensive error. Tone, which has no right answer, is judged by a model they validate. The COO's takeaway: ask which error each number guards against.
Quiz · Chapter 4
- "Recall" for a safety filter means:
- A safety/moderation filter should usually optimize:
- "99% accurate" on a problem where 2% of cases are harmful is suspicious because:
- The leadership question when a metric is chosen is: