Why Your Bot Makes Things Up
"It hallucinated" is the complaint you'll hear most. Half the time the fix isn't the model at all — it's whether you told the system to answer only from the documents, and to say "I don't know" when they don't cover it.
A language model is a fluent guesser: when the retrieved passages don't contain the answer, it will often produce something that sounds right rather than admit the gap. The defenses are boring and effective. Grounding: instruct the system to answer strictly from the retrieved passages, not its general knowledge. Citations: require it to point to the passage each claim came from, so a human — or an automated check — can verify. Abstention: make "I don't have that in the documents" a correct, rewarded answer rather than a failure.
That last one is where leaders get it wrong. If you push the team to raise the "answer rate," you're training the system to guess instead of abstain — manufacturing confident fabrications. In a regulated or high-trust product, a grounded "I don't know" is worth far more than a plausible invention.
The abstention debate
Sales wants the assistant to "always give an answer" because "I don't know" looks weak. Support wants it to abstain when unsure. Where do you land, and how do you frame the trade-off?
Land with support, and reframe it as risk, not politeness. "Always answer" doesn't produce more correct answers — it converts "I don't know" into confident guesses, some of which will be wrong and cited, which is the most damaging kind of error because users believe it. The right target isn't answer rate, it's correct-answer rate plus honest abstention. Tell sales: a bot that occasionally says "let me connect you to a human" builds trust; a bot that invents a policy loses a customer and maybe triggers a compliance issue.
A wealth app that invented a fee
A personal-finance assistant retrieved the right account documents and still told a user there was a "2.9% transfer fee" — a number that appeared nowhere in the retrieved passages. It was the model filling a gap with a plausible figure. In a financial product, a made-up fee isn't a cute error; it's a liability and a trust breach.
The team added two controls: the assistant now cites the passage behind every claim, and an automated check flags any sentence whose cited passage doesn't support it. Unsupported claims fell from 12% to 1.5%. The abstention rate rose — the bot now says "I don't see that in your account documents" more often — and the team counted that as a win. They'd rather abstain than invent, because one invented fee can cost more than a hundred honest "I don't knows."
Brightline sets the rule early: every answer must cite a document, and "that isn't covered in our docs" is a passing answer, not a bug. When a product manager asks to boost the answer rate, the eng lead pushes back — raising it would just mean more confident guesses. Brightline optimizes for grounded and honest instead.
Quiz · Chapter 5
- A bot "makes things up" most often because:
- The three main defenses are:
- Pushing to raise the "answer rate" tends to:
- "We told it in the prompt not to hallucinate" is: