Why "It Works in the Demo" Will Burn You
The single most expensive mistake in AI is trusting a demo. A demo is a handful of cherry-picked inputs on a system whose outputs change every time you run it. Evaluation is how you find out what happens on the other ten thousand.
Traditional software is deterministic: it passes a test or it doesn't. AI features aren't — the same prompt yields different answers, "correct" is replaced by "good enough," and quality is a distribution, not a fact. That means a single impressive output tells you almost nothing, and a single bad one isn't proof of failure either. The only trustworthy signal is an average over many realistic cases, with a sample size.
Why you should care: the gap between "demoed well" and "works at scale" is where launches slip, customers churn, and "the AI said something wrong" incidents are born. Evaluation is the discipline that closes that gap — and whether your team has it is a leading indicator of whether you can ship AI responsibly.
The "90% accurate" claim
An engineer tells you the new assistant is "90% accurate" and wants to launch Friday. You have sixty seconds. What three questions do you ask before you say yes?
Ask: (1) 90% on how many cases, and are they representative of real traffic? (90% on 20 hand-picked examples is noise; on 500 real ones it means something.) (2) 90% on which slices? A great average can hide a broken segment that's 50%. (3) How will we know on Monday if it's getting worse? (Is there any production monitoring, or are we flying blind after launch?) If they can't answer these, "90%" isn't a launch criterion — it's a vibe. Don't launch on a vibe.
Vero: shipped on a demo, churned on the truth
Vero launched a customer-facing AI assistant a week after a flawless leadership demo. Within a month the bill came due: on a sample of 400 real conversations, 34% of answers were wrong or unsupported. The demo questions had been the ones the team knew worked; real customers asked the messy edge cases nobody rehearsed.
The expensive part wasn't the wrong answers — it was the lost trust and the support tickets that rose instead of fell. A week of measuring on real conversations before launch would have surfaced the 34% at a fraction of the churn cost. The leadership lesson: a great demo tells you the happy path exists; it tells you nothing about the distribution your customers will actually hit.
One rollout runs through this book. Meridian, a Series-B fintech, is launching Remi, a billing support assistant. This chapter: the CEO watches Remi nail a demo and wants it live Friday. The eng lead asks one question — "on how many real tickets has this been measured?" — and the honest answer is "about eight." The launch slips two weeks. We'll follow the decisions to the end.
Quiz · Chapter 1
- A strong demo proves:
- The trustworthy signal of AI quality is:
- Hearing "it's 90% accurate," your first question is:
- The business reason evaluation matters is: