Why LLMs Broke Testing

For fifty years, testing software meant running the code and checking that the output matched what you expected. Put a language model in the loop and that check stops telling you anything useful. This chapter shows exactly how it fails, because everything else in the book is built on the replacement.

The test that stops working

Say you build a summarizer and write the test you've written a thousand times:

def test_summary():
    out = summarize("The cat sat on the mat in the warm afternoon sun.")
    assert out == "A cat rested on a mat in the sun."   # one blessed answer

Now run it five times:

All five summaries are fine, and the test passes once, by luck. Exact-match accuracy is 20%. What the test measures is whether the model happened to pick your wording. There are thousands of acceptable summaries of any paragraph, so you have to stop asking "is this the right answer?" and start asking "how good are the answers, on average, across many inputs?"

Your score is noisy too

Suppose you switch to a quality scorer that gives each case a number between 0 and 1. Twenty cases come back at 0.80. You change the prompt, run twenty cases again and get 0.85. It's tempting to call that progress, but a score over n cases has a margin of error. The rough 95% interval for a pass rate p is 1.96 × √(p(1−p)/n), and here's what that looks like at a few sizes:

At n=20, a 5-point gap sits well inside the noise. Engineers who come from deterministic testing tend to trip here, because they're used to thinking in individual test cases. Evaluation asks you to think in measurements, and a measurement always comes with a sample size.

Lab code · runs free

import random
from math import sqrt

TRUE_QUALITY = 0.80          # the system's real pass-rate (unknown in real life)

def simulated_eval(n):
    passes = sum(random.random() < TRUE_QUALITY for _ in range(n))
    p  = passes / n
    ci = 1.96 * sqrt(p * (1 - p) / n)        # 95% confidence interval
    return p, ci

random.seed(0)
for n in (20, 200, 2000):
    p, ci = simulated_eval(n)
    print(f"n={n:>4}:  score={p:.2f}   95% CI = +/- {ci:.3f}   "
          f"({p-ci:.2f} to {p+ci:.2f})")

n=  20:  score=0.85   95% CI = +/- 0.156   (0.69 to 1.01)
n= 200:  score=0.80   95% CI = +/- 0.055   (0.74 to 0.85)
n=2000:  score=0.81   95% CI = +/- 0.017   (0.79 to 0.82)