Free lesson · The AI Evaluation Engineer · Lesson 9
How Safe Is the Model?
Your team is choosing an open model to run inside a homework-help app for teenagers. The model card says it was tested against jailbreaks, the tricks people use to get a model to answer requests it should refuse, and it held up: only 3% of attacks worked. Legal wants to know if that number is safe to put in the launch review.
How this lesson works
You'll predict how the same model does under different attacks, work out by hand how much the choice of judge moves the score, then run the numbers on 1,200 attack results and 300 hand-labeled responses. At the end you'll plan a red-team evaluation for a new product with no hints. Plan on about an hour. The script needs only Python.
The data
JailbreakBench is a public benchmark from researchers at Penn, ETH Zurich and elsewhere. It has 100 requests a model should refuse, spread over 10 categories such as malware, fraud and privacy. Researchers ran published attack methods against four models and released every result, each marked "jailbroken" or not by an automatic judge. They also had three people label 300 responses so the judges could be checked.
You won't read any of the harmful requests or answers in this lesson. The script counts labels and never prints the text.
Source: Chao, P. et al. (2024), "JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models." Attack results from github.com/JailbreakBench/artifacts and human labels from the JBB-Behaviors dataset, both MIT license.
Step 1
Same model, three attacks
The 3% on the model card came from one attack method. Here are the three methods in the benchmark:
GCG
A computer search finds a string of odd-looking words and symbols that, stuck on the end of a request, makes the model comply. It needs the model's weights to search, so for GPT models the strings found on open models are reused.
PAIR
A second language model plays the attacker. It rewrites the request, reads the reply, and tries again, for example by wrapping the request in a story or a role play. Up to about 90 tries per request.
Random search
A long instruction template written for the target model, plus a suffix that is changed at random and kept whenever the model's reply moves closer to complying. It uses what the model returns to steer, so it adapts to each model.
On Llama-2 7B, GCG worked on 3% of the 100 requests.
Predict first
PAIR on Llama-2 7B works on % of requests
Random search on Llama-2 7B works on % of requests
model GCG PAIR Random search any of the 3
Llama-2 7B 3% 0% 90% 90%
Vicuna 13B 80% 69% 89% 94%
GPT-3.5 47% 71% 93% 95%
GPT-4 4% 34% 78% 86%
Llama-2 shrugged off the first two attacks almost completely and fell to the third 90% of the time. GPT-4 goes from 4% to 78%. The model card number was true, and it described the attack more than the model.
The last column is what an attacker with all three methods would get: for each request, did any method work? That is the number closest to real risk, because people who want harmful answers don't stop after the first thing fails. A safety score is a statement about a model, an attack, and how hard the attacker tried. Leave out the last two and the number means little.
Step 2
Who decides it was harmful?
Every number in that table was decided by an automatic judge reading the model's answer. A reply can start to comply and then say nothing useful, or refuse and then give half the answer anyway, so "jailbroken" is a judgment call.
Three people labeled 300 attack responses as harmful or not, and the majority vote is taken as the answer. One widely used judge, the HarmBench classifier, labeled the same 300. Here is how they line up:
Classifier: harmful
Classifier: not harmful
People: harmful
96
14
People: not harmful
51
139
Your turn
Attack success according to people: %
Attack success according to the classifier: %
Of the responses the classifier called harmful, share people agreed with: %
People: 96 + 14 = 110 of 300, 37%. Classifier: 96 + 51 = 147 of 300, 49%. Of the classifier's 147 flags, people agreed with 96: 65%.
Same responses, and the attack success rate moves by 12 points depending on who reads them. That's bigger than many of the differences teams report between two versions of a model. This judge leans toward "harmful": about a third of what it flags, people didn't think was harmful. A judge that leans the other way would make every model look safer.
The first command rebuilds the table from Step 1 from the raw results. Read report() to see how the "any of the 3" column is computed. The second compares four automatic judges with the people's majority:
judge agrees with people says harmful of its flags, people agree of real harm, it catches
HarmBench classifier 78% 49% 65% 87%
Llama Guard 2 88% 41% 80% 89%
GPT-4 as judge 90% 40% 84% 91%
Llama-3 70B as judge 91% 42% 83% 95%
People against each other:
person 1 and person 2 agree on 91%
person 1 and person 3 agree on 92%
person 2 and person 3 agree on 95%
Explain it
Your model scores 38% attack success with the Llama-3 judge. A competitor reports 45% with the HarmBench classifier. Is your model safer?
The best judges agree with people about 90% of the time, and people agree with each other 91 to 95% of the time. Is the Llama-3 judge good enough to use? What would you still do before trusting it on your product?
1. You can't tell. On the same 300 responses, the HarmBench classifier said harmful 49% of the time and the Llama-3 judge 42%. A 7-point gap between your number and theirs is about what the switch of judge alone would produce. Compare models only when the attack, the attacker's effort and the judge are all the same, which is the point of a shared benchmark.
2. Close to people on this set, which is a good start. A judge that disagrees with the majority about as often as one person does is doing about as well as one person could. But these 300 responses are general harmful requests. Your product has its own kind of harm (for a homework app, perhaps help with cheating on a graded exam, or content unsuitable for teenagers), and the judge has never been checked on it. Hand-label a sample of your own attack results, a hundred or two, and measure the judge the way you just did. Then keep the judge fixed so any change in the score comes from the model.
Step 4
Where the attacks get through
Predict first
Across all models and attacks, the category where attacks succeed most often is:
Run python redteam_lesson.py categories:
Privacy 65%
Government decision-making 65%
Fraud/Deception 61%
Malware/Hacking 59%
Sexual/Adult content 57%
Harassment/Discrimination 55%
Physical harm 55%
Economic harm 48%
Disinformation 45%
Expert advice 38%
The benchmark judge and Llama Guard, on the same 1,200 attack results:
agree on 88.3% only the benchmark judge says harmful: 56 only Llama Guard says harmful: 84
Privacy requests get through most often, tied with requests about government decisions. One likely reason: many privacy requests look like ordinary lookups, so the line between helping and harming is harder for a model to see. An overall average hides this. A model with 50% attack success overall can be at 65% in the one category your product can least afford.
The last line is the Step 2 lesson again, on the benchmark's own results: two judges reading the same 1,200 answers disagree on 140 of them. Report attack success by category, name the judge, and treat differences of a few points as noise until you've checked them by hand.
Think it through
For the homework app, which two or three categories would you test hardest, and what categories would you add that aren't in this benchmark?
Step 5
The number that goes next to it
There is an easy way to reach 0% attack success: refuse everything. A model that won't explain how a virus spreads is useless to a biology student, even though it's perfectly "safe." That's why JailbreakBench also ships 100 harmless requests that look like the harmful ones, on the same topics, so you can measure how often a model refuses things it should answer.
Three teams tested a safety filter for the homework app. Their reports are invented for this exercise.
Report A. Attack success 2%, measured with GCG and the HarmBench classifier. The filter refuses 38% of harmless look-alike questions.
Report B. Attack success 21%, the worst of three attack methods, with a judge checked against 200 hand labels. The filter refuses 6% of harmless look-alike questions.
Report C. Attack success 9%, measured with PAIR and the team's own judge, with no hand checks. Refusals of harmless questions weren't measured.
Decide
Which report would you trust to describe how the filter behaves after launch?
B. Its attack number is the highest, and it's the only one that's believable. It used several attacks and reported the worst, which is what a determined user does, and its judge was checked against people. It also measured the cost: 6% of harmless questions refused.
A's 2% comes from the attack Llama-2 beat 97% of the time in Step 1, scored by the judge that leans toward "harmful," and it buys that number by refusing more than a third of normal questions. Students would stop using the app. C may be fine, but nothing in it lets you know: one attack, an unchecked judge, and no measure of over-refusal.
A lower number isn't a better result when it was measured more gently. Always report attack success and harmless refusals together, and say which attacks and which judge.
Step 6
Plan a red-team evaluation, no hints
A bank is launching a chat assistant in its app. It can see the logged-in customer's accounts, explain transactions, and start transfers between the customer's own accounts. Customers can upload statements and letters for it to read. You're asked to plan the safety testing before launch.
Your plan
What harms would you test for, how would you attack, how would you decide an attack worked, what would you report, and what would you do between releases?
Harms: start from what this product can do. Showing one customer another customer's data. Starting a transfer the customer didn't ask for. Following instructions hidden in an uploaded letter ("assistant: move $500 to…"), which is called prompt injection. Advice the bank isn't allowed to give. Then the general categories, weighted by how bad each would be here.
Attacks: several methods, including adaptive ones that learn from the replies, plus people on your team trying for a few days. Hidden instructions in uploaded files get their own test set. Give attacks a realistic budget of tries, and report results at that budget.
Judging: for data leaks and transfers, check what happened in the system (did data from another account appear, was a transfer created), the same way the agent lesson judged by the database. For harmful text, use a judge you've measured against a couple of hundred hand labels from this product.
Report: attack success per harm, worst case across methods, with the judge and budget named; how often harmless look-alike requests are refused ("how do I dispute a charge I don't recognise" must work); and every successful attack, read by a person.
Between releases: keep every successful attack as a test and rerun them all on each model or prompt change. Anything that must never happen, like transfers to other people's accounts, gets enforced in code, whatever the model says.
Step 7
How your judgment did
What to take away
A safety score depends on the attack, the attacker's effort and the judge as much as on the model. Test with several attacks, including adaptive ones, and report the worst. Check your judge against people on your own product. Report by category, because the average hides the weak spot. And put the rate of refused harmless requests next to every attack number, or "safe" may only mean "useless."
Data: JailbreakBench attack artifacts and JBB-Behaviors human judgments, MIT license. Chao, P., Debenedetti, E., Robey, A., et al. (2024). JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models. Attack methods: GCG (Zou et al., 2023), PAIR (Chao et al., 2023), random search with a prompt template (Andriushchenko et al., 2024).