Free lesson · The AI Evaluation Engineer · Lesson 6
Did the Model Make That Up?
Your company's assistant answers questions from documents it retrieves, the setup called retrieval-augmented generation (RAG). The pitch is that it only says what the sources say. You've been asked to check that promise, compare models, and decide whether a cheap automatic check could catch the answers that go beyond their sources before customers see them.
How this lesson works
You'll check five real answers against their sources yourself, predict how often six models make things up, measure it on nearly 18,000 answers, and test two automatic checks. Plan on about an hour.
The data
RAGTruth is a research dataset of 17,790 answers from six language models: GPT-3.5, GPT-4, three sizes of Llama 2, and Mistral 7B. Every model was given source documents and asked to work only from them, on three kinds of task: answering a question from web passages, summarizing a news article, and describing a business from its listing data. Annotators then marked, word by word, anything an answer said that the sources don't support or that contradicts them, and wrote a note explaining each mark.
In this lesson, "made up" means exactly that: not supported by the sources the model was given. A statement can be true in the world and still count, and you'll see why that definition matters.
Source: Niu et al., 2024, "RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models." Dataset: github.com/ParticleMedia/RAGTruth, MIT license. Answers and sources are shown unedited.
Step 1
Check five answers against their sources
For each answer, read the question and the sources, then the answer. Decide whether it says anything the sources don't support, and if so, copy the phrase. Take your time: this is exactly the work the annotators did, and it's slower than it looks.
Stop and think
Notice the different kinds of mistakes: a wrong number that looks right (the weather), a confident specific detail that appears nowhere (the tax line), a flat contradiction of a source (the butterflies), and a sentence that's true in the real world but isn't in the sources (the drug law). The last one is the hardest call. If your product promises "only what the documents say," it counts; if you only care about truth, it may not. Decide which rule your product needs before anyone labels anything.
Step 2
How often, and which model?
Predict first
Every model was given the sources and told to use them. What share of each model's answers contain at least one made-up or contradicting passage?
Read load() first, especially how an answer is counted as "made up." You should see:
model all tasks QA Summary Data2txt
gpt-3.5-turbo-0613 11% (10% to 13%) 5% 4% 24%
gpt-4-0613 12% (11% to 14%) 3% 7% 26%
llama-2-70b-chat 42% (41% to 44%) 25% 20% 79%
llama-2-13b-chat 52% (50% to 54%) 30% 29% 94%
llama-2-7b-chat 57% (55% to 59%) 43% 43% 82%
mistral-7B-instruct 61% (60% to 63%) 30% 63% 91%
all models 39%
Explain it
GPT-4 is at 12% and GPT-3.5 at 11%. Is GPT-4 worse here?
Even GPT-4 goes wrong on about one business description in four. Given the data a business listing contains (name, hours, a list of attributes like parking and Wi-Fi, some reviews), what do you think goes wrong?
1. No difference you can claim. The two ranges, 10% to 13% and 11% to 14%, overlap almost entirely. What you can say is that both are far below the open models from 2023, and that "retrieval stops made-up answers" is false for all of them.
2. Empty fields and summarized details. Run python madeup_lesson.py nulls. Business listings have many fields that are empty (null), meaning nobody recorded the information. Models often turn an empty field into a "no": "does not offer Wi-Fi" when the listing says nothing about Wi-Fi either way. Sometimes they fill it in the other direction, calling a place "trendy" or "open seven days a week" with nothing behind it. The script finds that 19% of the marked passages in business descriptions come with an annotator note about an empty field. Models also compress opening hours incorrectly, turning "closed Mondays" into "open every day." Structured data tempts a model to fill every gap.
Step 4
Can a cheap check catch them?
Reading every answer isn't possible at scale. A tempting shortcut: flag any answer that contains a number, or a capitalized name, that doesn't appear in its sources. Invented facts often involve numbers and names, so this should catch a lot. Look at unsupported_numbers() and unsupported_names() in the script.
Predict first
The numbers only check will catch what share of made-up answers? %
Of all the passages annotators marked, what share contain a digit? %
Run python madeup_lesson.py detect:
check answers flagged caught (recall) flags that were real (precision)
numbers only all 17,790 23% 28% 47%
numbers only GPT-4 only 11% 27% 29%
numbers only Mistral only 19% 23% 75%
numbers or names all 17,790 70% 85% 48%
numbers or names GPT-4 only 61% 86% 17%
numbers or names Mistral only 70% 80% 71%
Of 12,361 marked passages, 22% contain a digit.
The numbers-only check catches just 28% of made-up answers, because only 22% of the marked passages contain a number at all. Most invented content is words: a wrong claim, an added detail, a contradiction. Adding names catches 85%, but by flagging 70% of all answers, since paraphrases and ordinary capitalized words don't appear word for word in the sources. A check that flags most answers just hands people a long queue to read.
Step 5
The same check, two models
Look at the "numbers or names" rows again. The check itself is identical, but on GPT-4 only 17% of its flags are real problems, and on Mistral 71% are.
Your turn
Your product runs GPT-4 and answers 1,000 questions a day. Use the GPT-4 row of the "numbers or names" check: it flags 61% of answers, and 17% of flags are real.
Answers flagged per day:
Of those, real problems:
Made-up answers per day that the check misses (about 12% of 1,000 are made up; it catches 86%):
It flags about 610 answers a day, of which about 104 are real problems (610 × 0.17), and it misses about 17 (120 made up, 86% caught). So a reviewer reads 610 answers to find 104, and 17 still reach customers.
Why is precision so different between models? Precision depends on how common the problem is. When a model makes things up in 61% of answers, most flags land on real problems. When it does so in 12%, most flags land on good answers. The same check that looks useful on a weak model looks useless on a strong one. Whenever you report a detector's precision, report the base rate it was measured at, and re-measure it on the model you'll run in production.
Step 6
A new product, no hints
A travel company wants to generate short descriptions of restaurants from their listing data: name, address, hours, a set of attributes (parking, Wi-Fi, outdoor seating, noise level and so on, many of them empty) and a few reviews. The descriptions will be published on the company's site. The plan is to use GPT-4.
Your recommendation
Using what you measured, write what could go wrong and what you'd put in place before launch.
What could go wrong. This is the business-description task, where even GPT-4 made something up in 26% of answers. Expect about one published description in four to state something the listing doesn't support, most often about empty fields ("no Wi-Fi," "open every day") or garbled hours. For a travel site, those are the details customers act on.
Before launch:
Make empty fields explicit. Remove them before generation, or pass them as "unknown," and instruct the model to mention only attributes that are present.
Check the structured claims in code. Hours, prices and yes/no attributes are fields in the input, so a script can compare what the description says about them with the listing. Unlike free text, this data makes a precise automatic check possible.
Don't rely on a word-matching check like the one in Step 4. At GPT-4's error rate, most of its flags would be false alarms.
Measure before and after. Have people label a few hundred generated descriptions against their listings, the way RAGTruth's annotators did, to get your own rate. Launch only when it's below a limit the business agrees to, and keep sampling after launch.
Step 7
How your judgment did
What to take away
Giving a model sources doesn't stop it from making things up; measure how often it does, by model and by task. Decide what "made up" means for your product before anyone labels. Most invented content is in the words, with few numbers to check, so simple checks miss most of it or flag almost everything. A detector's precision depends on how common the problem is, so measure it on your own model. And where the input is structured, check the output against it in code.
Data: RAGTruth, Niu, C. et al. (2024). RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. ACL. github.com/ParticleMedia/RAGTruth, MIT license.