Why LLMs Broke Testing

Fifty years of software testing rests on one move: run the code, assert the output equals the expected value. The moment a language model enters the loop, that move stops working — and understanding exactly why is the foundation of the whole job.

Let's make it concrete. You build a summarizer and write the test you've written ten thousand times:

The test that betrays you

def test_summary():
    out = summarize("The cat sat on the mat in the warm afternoon sun.")
    assert out == "A cat rested on a mat in the sun."   # one blessed answer

Run it five times:

Five good summaries. Exact-match accuracy: 20% — and 0% if run 5 hadn't matched by luck. The metric measures phrasing, not quality. There are thousands of equally good summaries, so "correct" is replaced by "good," and good is a distribution, not a point.

The deeper trap: even your score is noisy

So you ditch exact match for a quality scorer (0–1 per case). You run 20 cases: 0.80. Next week, new prompt, 20 cases: 0.85. You ship a "5-point win." You just fooled yourself. A score over n cases has a margin of error — the rough 95% interval is 1.96 × √(p(1−p)/n):

At n=20 your "win" is smaller than the noise. This is the shift that trips up every engineer from deterministic testing: you stop thinking in test cases and start thinking in measurements with sample sizes. A score without an n is a rumor.

Lab code — runs free

import random
from math import sqrt

TRUE_QUALITY = 0.80          # the system's real pass-rate (unknown in real life)

def simulated_eval(n):
    passes = sum(random.random() < TRUE_QUALITY for _ in range(n))
    p  = passes / n
    ci = 1.96 * sqrt(p * (1 - p) / n)        # 95% confidence interval
    return p, ci

random.seed(0)
for n in (20, 200, 2000):
    p, ci = simulated_eval(n)
    print(f"n={n:>4}:  score={p:.2f}   95% CI = +/- {ci:.3f}   "
          f"({p-ci:.2f} to {p+ci:.2f})")

n=  20:  score=0.85   95% CI = +/- 0.156   (0.69 to 1.01)
n= 200:  score=0.80   95% CI = +/- 0.055   (0.74 to 0.85)
n=2000:  score=0.81   95% CI = +/- 0.017   (0.79 to 0.82)