Why the "Chat With Your Docs" Demo Lies
Nothing sells a project like watching an AI answer a hard question from your own handbook. Nothing predicts a failed launch like trusting that demo. The demo is three questions someone already knew would work; production is the other ten thousand.
A RAG system — "retrieval-augmented generation," the technology behind every "chat with your docs" feature — works by finding the relevant passages in your documents and handing them to a language model to answer from. That means it has two ways to fail that a demo never shows: it can fail to find the right passage, or it can find it and answer wrong anyway. In a demo, the questions were chosen because retrieval happens to work. Real users ask the questions you didn't rehearse.
The only trustworthy signal is the same one in every field of measurement: performance across many realistic questions, with a sample size — not one dazzling answer, and not one embarrassing screenshot either. The gap between "demoed well" and "answers our customers correctly" is exactly where support tickets, refunds, and "the AI told a customer the wrong policy" incidents are born.
The Friday launch
Your team demos a doc assistant that nails five questions and asks to launch Friday. You have sixty seconds. What three questions do you ask before you say yes?
Ask: (1) On how many real questions has this been measured, and were they representative of what customers actually ask? Five is a demo; 300 sampled from real traffic is a signal. (2) When it's wrong, is it because it couldn't find the answer or because it found it and mis-answered? Those are different fixes and a healthy team can tell them apart. (3) After launch, how will we know on Monday if it's getting worse? If they can't answer these, "it looked great" isn't a launch criterion — it's a hope.
PayString: shipped on a demo, churned on the truth
PayString, a payments startup, launched a customer-facing help assistant a week after a flawless board demo. Within a month, support saw the fallout: on a sample of 400 real conversations, 34% of the assistant's answers either cited nothing from the docs or contradicted them. The demo questions had been about their flagship feature, which was thoroughly documented; real customers asked about edge cases, recent changes, and error messages — exactly the content retrieval handled worst.
The expensive part wasn't the wrong answers; it was the trust. Customers who got one confidently wrong answer stopped believing the assistant entirely, and support volume went up, not down. A week of measuring on real questions before launch would have caught the 34% — and cost a fraction of the churn.
At Brightline, a mid-size SaaS company, the CEO sees one perfect demo of an internal "chat with our docs" tool and wants it live by Friday. The eng lead asks a single question — "on how many real employee questions have we measured this?" — and the honest answer is "about eight." Launch slips two weeks; we'll follow the rollout from here.
Quiz · Chapter 1
- A great "chat with your docs" demo proves:
- A RAG system has two failure modes a demo hides:
- The trustworthy signal of quality is:
- The business reason to measure before launch is: