Free lesson · The AI Evaluation Engineer · Lesson 3
Can an Automatic Score Tell a Good Summary From a Bad One?
Your team builds a feature that summarizes news articles. Reading every summary by hand is too slow, so someone proposes the standard automatic score, ROUGE, which compares a summary's words with summaries written by people. Before you wire it into your tests, you'll find out what it measures and what it misses, using 1,600 real summaries that experts rated by hand.
How this lesson works
You'll judge four summaries yourself, compute ROUGE by hand for one of them, then test ROUGE against the experts on the whole dataset and decide where it belongs in a real pipeline. Plan on about an hour. Step 2 needs a calculator.
The data
SummEval is a research dataset built from 100 CNN and Daily Mail news articles. Sixteen different summarization systems each summarized every article, and people wrote 11 more summaries per article as references. Experts then rated all 1,600 machine summaries from 1 to 5 on four things: relevance (does it capture the important points?), coherence (is it well organized?), fluency (is each sentence well written?) and consistency (does it only say things the article supports?). The machine summaries are shown exactly as the systems produced them, which is why many are in lowercase with spaces before punctuation.
Source: Fabbri et al., 2021, "SummEval: Re-evaluating Summarization Evaluation." Dataset: mteb/summeval on Hugging Face, MIT license. Articles are from the CNN/Daily Mail news dataset.
Step 1
Judge four summaries yourself
Read the article below. It's copied as it appears in the dataset, so image captions run into the text; that's how real scraped news looks.
Now read four machine summaries of it. For each, give it a rank from 1 (best) to 4 (worst), and mark whether it contains a factual error, meaning anything the article doesn't support.
One more prediction
Which summary will get the highest ROUGE score?
The experts and ROUGE rank these four in almost opposite order. B has the highest ROUGE score and the second-lowest expert rating. Read B again: "po-faced teenagers next to their daughters." The article says teenagers are pictured next to their fathers. B also ends mid-sentence: "the 37-year-old's images include." D has the same problem, with teenagers "in the air next to their daughters." Meanwhile C, a short, vague summary that the experts rated mostly accurate (4.33 for consistency), gets the lowest ROUGE score.
Step 2
Compute ROUGE by hand
ROUGE-1 counts words that the summary and a reference summary share. Here is summary B, split into words the way the score sees them: lowercase, punctuation removed.
And here is one of the human reference summaries: "A series of photos sees Japanese dads jumping next to their daughters. They are part of a new book by Japanese photographer Yûki Aoyama. The book's title roughly translates as Daughter and Salary Man."
The counting rule. A word counts as shared as many times as it appears in both lists, whichever is fewer. "of" appears twice in B and twice in the reference, so it counts twice. "images" appears three times in B and not at all in the reference, so it counts zero times.
23 shared words. Precision 23 ÷ 41 = 0.56, recall 23 ÷ 35 = 0.66, and ROUGE-1 = 0.61. (Scoring against all 11 references and keeping the best match, the usual practice, gives the same 0.61 here.)
Look at what you counted. "teenagers" isn't in the reference, but "next," "to," "their" and "daughters" all are, so the false phrase "teenagers next to their daughters" earns four matching words. ROUGE has no idea whether the words are arranged into a true statement. It rewards reusing the right words. A system that copies phrases from the article and stitches them together wrongly can score better than one that writes an accurate summary in its own words.
Step 3
Test ROUGE against the experts
One article can be a fluke. Download rouge_lesson.py, put it in an empty folder, and run:
pip install pandas pyarrow
python rouge_lesson.py
Read rouge1() first: it's the calculation you just did by hand. The script scores all 1,600 summaries and measures how well ROUGE's ordering matches the experts', using a Spearman correlation: 1 means the same order, 0 means no relation, and 0.3 is weak. It also checks one more "score": the summary's length in words. Before you run it:
Predict first
Correlation between ROUGE-1 and the experts' consistency ratings, one summary at a time:
Will ROUGE agree with the experts better when judging single summaries, or when comparing whole systems averaged over 100 articles?
One summary at a time (all 1,600 summaries)
relevance coherence fluency consistency
rouge1 0.32 0.21 0.12 0.17
rouge2 0.22 0.13 0.08 0.14
words 0.25 0.09 0.01 0.09
Whole systems (each system's average over 100 articles, 16 systems)
relevance coherence fluency consistency
rouge1 0.75 0.52 0.60 0.46
rouge2 0.66 0.40 0.57 0.28
words 0.21 -0.04 0.32 0.62
Share of summaries the experts rated 3 or lower on consistency: 10%
Explain it
For single summaries, ROUGE-1 agrees with the experts' relevance ratings at 0.32. Summary length alone gets 0.25. What does that tell you about ROUGE?
Why would ROUGE work so much better for comparing whole systems than for judging one summary?
For whole systems, length agrees with consistency (0.62) better than ROUGE does (0.46). Why might longer summaries be more factually consistent?
1. Mostly a length-and-overlap signal. A score that's only slightly better than counting words isn't judging quality in any deep sense. Longer summaries share more words with the references, so they score higher, whether or not the extra words are right.
2. Averaging cancels noise. On any single summary ROUGE is often wrong, as you saw in Step 1. But its errors go in both directions, and averaged over 100 articles they largely cancel, leaving the real differences between systems visible. This is the same reason averages over more items were more reliable in Lesson 1. A score can be useless for grading one output and still useful for comparing systems.
3. Copying is safe. Some of these systems mostly copy whole sentences from the article; others write new sentences. Copied sentences are longer and rarely state anything false, because they're the article's own words. Systems that write their own sentences are shorter and more likely to garble facts, like B in Step 1. So length here stands in for "how much was copied," which tracks consistency. That's a property of these 16 systems, and other systems may behave differently.
Step 4
Find the error ROUGE rewarded
Run:
python rouge_lesson.py cases
It lists summaries that are in their article's top three by ROUGE but that the experts rated 2 or lower on consistency, with the start of each article. The first one is about an Australian politician, Bob Katter. Read the summary and the article opening carefully.
Your turn
What is factually wrong in the first summary, and why would it matter if your product published it?
The summary says: "mr katter said he would quit after he was accused of domestic violence." The article says Katter claimed the government believed another MP, Billy Gordon, would quit after Gordon was accused of domestic violence. The summary moves an accusation of domestic violence onto the wrong person, by name. Published by a news product, that's a correction at best and a defamation complaint at worst.
It scored 0.67, the second-highest ROUGE of the 16 summaries of that article, because nearly every word in it is in the references. This is the most important limit of word-overlap scores: the errors they miss are exactly the errors that do the most damage.
Step 5
Use ROUGE to choose a summary?
A teammate proposes generating several candidate summaries per article and using ROUGE to pick the one to show. You can test that on this data: for each article, pick one of the 16 summaries in different ways and look at the experts' ratings of what gets picked.
Predict first
Average expert relevance of a random pick is 3.78; of the experts' own favorite, 4.70. The ROUGE pick will average about
Picking the longest summary will do
Run python rouge_lesson.py bestof. You should see:
how the pick was made relevance consistency picks rated 3 or lower on consistency
at random (average of all) 3.78 4.66 10%
highest ROUGE-1 4.03 4.87 4%
longest summary 3.96 4.75 7%
experts' favorite 4.70 4.99 0%
ROUGE helps a little. It lifts relevance from 3.78 to 4.03 and cuts inconsistent picks from 10% to 4%. But it closes only about a quarter of the gap to the experts' choice, and picking the longest summary gets most of the same benefit. And one pick in 25 is still a summary the experts rated as inconsistent. If you show 10,000 summaries a day, that's about 400 with errors.
Step 6
Where does ROUGE belong?
Here are four places your team could use ROUGE. For each, decide whether it's a good use, and if not, what you'd use instead. There's one fact about production you'll need: ROUGE requires human-written reference summaries to compare against.
Your decisions · no hints
Use
Your call
Nightly check comparing summarizer v1 and v2 on 500 articles with reference summaries:reasonable, with a partner. This is the system-level comparison where ROUGE agrees with experts at 0.75 on relevance. Its agreement on consistency is much weaker (0.46), so add a separate factual consistency check, and have someone read a sample of the summaries whose scores changed most.
Blocking any single summary whose ROUGE falls below a threshold:no. At the level of one summary, ROUGE agrees with experts on consistency at only 0.17, and Step 4 showed it ranks harmful errors near the top. It would block good summaries and pass bad ones. And in production there's no reference summary to compare against, so it can't even be computed.
Picking the best of several candidate summaries:weak. Step 5 showed a small gain, most of which you'd get by picking the longest, and it needs references you won't have in production. A model-based judge checked against human ratings, the subject of the judge lesson, is the better tool.
Monitoring for made-up facts in production:no. This is exactly what ROUGE misses, and again there are no references. Use a factual consistency check that compares the summary with the article itself, plus regular human review of a sample.
The pattern: overlap scores are cheap and useful for comparing systems on a fixed test set with references. They are not quality judges for individual outputs, and they can't run where there's no reference, which is everywhere in production.
Step 7
How your judgment did
What to take away
Know what a score counts before you trust it: ROUGE counts shared words and is blind to whether they form a true statement. Check any automatic score against human ratings, both on single outputs and on system averages, because the answers differ. Compare it with a trivial baseline like length. Read the cases where the score and people disagree most. And check whether the score can even run where you want to use it.