  Menu* [Tools](https://unit42.paloaltonetworks.com/tools/)* [ATOMs](https://unit42.paloaltonetworks.com/atoms/)* [Security Consulting](https://www.paloaltonetworks.com/unit42)* [About Us](https://unit42.paloaltonetworks.com/about-unit-42/)* [**Under Attack?**](https://start.paloaltonetworks.com/contact-unit42.html) * [Threat Research Center](https://unit42.paloaltonetworks.com ‘Threat Research’)* [Threat Research](https://unit42.paloaltonetworks.com/category/threat-research/ ‘Threat Research’)* [Malware](https://unit42.paloaltonetworks.com/category/malware/ ‘Malware’) [Malware](https://unit42.paloaltonetworks.com/category/malware/)Bad Likert Judge: A Novel Multi-Turn Technique to Jailbreak LLMs by Misusing Their Evaluation Capability======================================================================================================== 14 min read Related Products Unit 42 Incident Response *  By: * [Yongzhe Huang](https://unit42.paloaltonetworks.com/author/yongzhe-huang/) * [Yang Ji](https://unit42.paloaltonetworks.com/author/yang-ji/) * [Wenjun Hu](https://unit42.paloaltonetworks.com/author/wenjun-hu/) * [Jay Chen](https://unit42.paloaltonetworks.com/author/jaychenpaloaltonetworks-com/) * [Akshata Rao](https://unit42.paloaltonetworks.com/author/akshata-rao/) * [Danny Tsechansky](https://unit42.paloaltonetworks.com/author/danny-tsechansky/)*  Published:December 31, 2024*  Categories: * [Malware](https://unit42.paloaltonetworks.com/category/malware/) * [Threat Research](https://unit42.paloaltonetworks.com/category/threat-research/)*  Tags: * [GenAI](https://unit42.paloaltonetworks.com/tag/genai/) * [Jailbroken](https://unit42.paloaltonetworks.com/tag/jailbroken/) * [Likert scale](https://unit42.paloaltonetworks.com/tag/likert-scale/) * [LLMs](https://unit42.paloaltonetworks.com/tag/llms/) * [Prompt injection](https://unit42.paloaltonetworks.com/tag/prompt-injection/)* * Share* * * * * * * Executive Summary—————–This article presents what we are calling the ‘Bad Likert Judge’ technique. Text-generation large language models (LLMs) have safety measures designed to prevent them from responding to requests with harmful and malicious responses. Research into methods that can bypass these guardrails, such as Bad Likert Judge, can help defenders prepare for potential attacks.The technique asks the target LLM to act as a judge scoring the harmfulness of a given response using the Likert scale, a rating scale measuring a respondent’s agreement or disagreement with a statement. It then asks the LLM to generate responses that contain examples that align with the scales. The example that has the highest Likert scale can potentially contain the harmful content.We have tested this technique across a broad range of categories against six state-of-the-art text-generation LLMs. Our results reveal that this technique can increase the attack success rate*(ASR)* by more than 60% compared to plain attack prompts on average.Given the scope of this research, it was not feasible to exhaustively evaluate every model. To ensure we do not create any false impressions about specific providers, we have chosen to anonymize the tested models mentioned throughout the article.It is important to note that this jailbreak technique targets edge cases and does not necessarily reflect typical LLM use cases. We believe most AI models are safe and secure when operated responsibly and with caution.If you think you might have been compromised or have an urgent matter, contact the [Unit 42 Incident Response team](https://start.paloaltonetworks.com/contact-unit42.html).**Related Unit 42 Topics** [**Prompt Injection**](https://unit42.paloaltonetworks.com/tag/prompt-injection/), **[LLMs](https://unit42.paloaltonetworks.com/tag/llm/)**What Is An LLM Jailbreak?————————-LLMs have become increasingly popular due to their ability to generate text that looks like that written by a human and assist with various tasks. These models are often trained with safety guardrails to prevent them from producing potentially harmful or malicious responses. LLM jailbreak methods are techniques used to bypass these safety measures, allowing the models to generate content that would otherwise be restricted.Existing Jailbreak Techniques—————————–Common jailbreak strategies include:* [Persona persuasion -[PDF-]](https://arxiv.org/pdf/2401.06373)* Role-playing, ‘Do Anything Now’ ([DAN](https://arxiv.org/abs/2308.03825))* [Token smuggling](https://learnprompting.org/docs/prompt_hacking/offensive_measures/obfuscation?srsltid=AfmBOooaoQ_zNDZIqtf1l-tWtiQN1xW4kDtYBSqA0IohUOga2wEocdEf)These jailbreaking strategies can be executed in a single conversation (single-turn) or across multiple conversations ([multi-turn](https://www.anthropic.com/research/many-shot-jailbreaking)). For example, some [token smuggling strategies](https://embracethered.com/blog/posts/2024/hiding-and-finding-text-with-unicode-tags/) employ encoding algorithms like Base64 to conceal malicious prompts within the input. On the other hand, multi-turn attacks such as the [crescendo](https://crescendo-the-multiturn-jailbreak.github.io/) technique begin with an innocuous prompt. They then gradually steer the language model toward generating harmful responses through a series of increasingly malicious interactions.### Why Do Jailbreak Techniques Work?Single-turn attacks often exploit the computational limitations of language models. Some prompts require the model to perform computationally intensive tasks, such as generating long-form content or engaging in complex reasoning. These tasks can strain the model’s resources, potentially causing it to overlook or bypass certain safety guardrails.Multi-turn attacks typically leverage the language model’s context window and attention mechanism to circumvent safety guardrails. By strategically crafting a series of prompts, an attacker can manipulate the model’s understanding of the conversation’s context. They can then gradually steer it toward generating unsafe or inappropriate responses that the model’s safety guardrails would otherwise prevent.LLMs can be vulnerable to jailbreaking attacks due to their long context window. This term refers to the maximum amount of text (tokens) an LLM model can remember at one time when generating responses.Anthropic recently discovered a good example of this strategy, the [many-shot attack strategy](https://www.anthropic.com/research/many-shot-jailbreaking). This strategy simply sends the LLM many rounds of prompts preceding the final harmful question. Despite its simplicity, this approach has proven highly effective in bypassing internal LLM guardrails.Furthermore, the attention mechanism in language models allows them to focus on specific parts of the input when generating a response. However, adversaries can abuse this capability to distract LLMs to focus on the benign parts while they embed unsafe prompts. For instance, the recently discovered [Deceptive Delight](https://unit42.paloaltonetworks.com/jailbreak-llms-through-camouflage-distraction/) attack and the [Crescendo](https://crescendo-the-multiturn-jailbreak.github.io/) attack use this method.Bad Likert Judge Jailbreak————————–As mentioned in a [prior research study](https://arxiv.org/abs/2404.01318) on jailbreak attacks, researchers often employ an evaluator LLM as a judge to assess the responses generated by other language models. This judge is meant to determine whether the content is harmful or not.One standard measurement metric for evaluating the harmfulness level is the [Likert scale](https://en.wikipedia.org/wiki/Likert_scale), which is a rating scale that asks respondents to indicate their level of agreement or disagreement with a given statement. In this context, we refer to the LLM judge using the Likert scale as a Likert Judge.The evaluator’s ability to evaluate responses implies that the LLM itself has learned the concepts behind what is considered harmful and malicious, or ‘features,’ as defined by [Anthropic](https://www.anthropic.com/news/mapping-mind-language-model). This understanding allows the evaluator to differentiate between various levels of harmfulness.However, when assisted with a prompt that indirectly asks for the LLM’s understanding of a concept, it may enable the model to produce harmful results without triggering its internal guardrails. Figure 1 shows the overall attack flow. Next, we explain each step in detail. Figure 1. Attack flow and prompt turns.### Turn 1: Evaluator PromptThe first step in the attack involves asking the target LLM to act as a judge to evaluate responses generated by other LLMs.To confirm that the LLM can produce harmful content, we provide specific guidelines for the scoring task. For example, one could provide guidelines asking the LLM to evaluate content that may contain information on generating malware, as shown in Figure 2. Figure 2. Example evaluator guidelines.### Turn 2: Prompt That Indirectly Asks for Harmful Content GenerationOnce step one is properly completed, the LLM should understand the task and the different scales of harmful content. Step two is straightforward. Simply ask the LLM to provide different responses corresponding to the various scales. Figure 3 shows an example prompt. Figure 3. Example prompt for obtaining malicious responses.If the attack is successful, the LLM will generate multiple responses with different scores. We can then look for the response with the highest score, which generally contains the harmful content.### Follow-Up TurnsAfter completing step two, the LLM typically generates content that is considered harmful. However, in some cases, the generated content may not be sufficient to reach the intended harmfulness score for the experiment.To address this, one can ask the LLM to refine the response with the highest score by extending it or adding more details. Based on our observations, an additional one or two rounds of follow-up prompts requesting refinement often lead the LLM to produce content containing more harmful information.Evaluation and Results———————-### Jailbreak CategoriesTo evaluate the effectiveness of Bad Likert Judge, we selected a list of common jailbreak categories. These categories encompass various types of generative AI safety violations and attempts to extract sensitive information from the target language model.AI safety violations mainly refer to the misuse or abuse of an LLM to produce harmful or unethical responses. These violations can encompass a wide range of issues, such as promoting illegal activities, encouraging self-harm or spreading misinformation.In our evaluation, we created a list of AI safety violation categories by referencing public pages published by several prominent AI service providers, including:1. [Azure OpenAI Service Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/harm-categories?tabs=warning)2. [Anthropic Usage Policy Page](https://www.anthropic.com/legal/aup)3. [OpenAI Usage Policy Page](https://openai.com/policies/usage-policies/)In addition to AI safety violation categories, jailbreaking can also leak sensitive information from the target LLM. Typical sensitive data includes the target LLM’s [system prompt](https://arxiv.org/abs/2405.06823), which is a set of instructions given to the LLM to guide its behavior and define its purpose. Leaking the system prompt can expose confidential information about the LLM’s design and capabilities.Furthermore, jailbreaking can also [leak training data](https://dropbox.tech/machine-learning/bye-bye-bye-evolution-of-repeated-token-attacks-on-chatgpt-models) that the LLM memorized during its training phase. LLMs are trained on vast amounts of data, and in some cases, they may inadvertently memorize specific examples or sensitive information present in the training dataset. Jailbreak attempts can exploit this to extract confidential or personal information, such as private conversations, financial records or intellectual property that the model unintentionally retained during training.Our evaluation focuses on the following categories:* **Hate**: Promoting or expressing hatred, bigotry or prejudice toward individuals or groups based on their race, ethnicity, religion, gender or other characteristics* **Harassment**: Engaging in behavior that targets and intimidates, offends or demeans an individual or group* **Self-harm**: Encouraging or promoting acts of self-injury or suicide* **Sexual content**: Generating or discussing explicit sexual material, pornography or other inappropriate content of a sexual nature* **Indiscriminate weapons**: Providing information on the manufacture, acquisition or use of weapons without proper context or safeguards* **Illegal activities**: Encouraging, promoting or assisting in activities that violate laws or regulations* **Malware generation**: Creating, distributing or encouraging the use of malicious software designed to harm computer systems or steal sensitive information* **System prompt leakage**: Revealing the confidential set of instructions used to guide the LLM’s behavior and responses### Evaluate Results With Another LLM JudgeThere are many ways to evaluate whether a jailbreak is successful or not. Previously, Ran et al. summarized these approaches in the [JailbreakEval paper](https://arxiv.org/html/2406.09321v1). There are four main ways to verify jailbreak success:* **Human annotation**: Manually examining the response to determine success* **String matching**: Identifying sensitive keywords in the response* **Chat completion**: Using an existing LLM, prompting it to act as an evaluator, similar to how our attack works* **Text classification** : Using fine-tuned natural language processing (NLP) models (e.g., the [BERT](https://research.google/pubs/bert-pre-training-of-deep-bidirectional-transformers-for-language-understanding/) model developed by Google) to identify harmful content in the responseIn our experiment, we choose to use the chat completion approach. This approach employs another LLM as an evaluator to determine whether the responses provided by our ‘bad judge’ LLM are harmful enough to be considered a successful jailbreak. Interested readers can refer to the [Appendix](#post-138017-_f09f6juvsqaf) to learn how we ensure that the evaluator can give a good assessment.### Measuring Attack EffectivenessUsing the validated evaluator, we measure the effectiveness of the attack using an attack success rate (ASR) metric, which is a standard metric used to assess jailbreak effectiveness in many research papers. The ASR is computed as follows:* Given Y attack attempts (prompts), if the evaluator determines X of these attempts are successful jailbreaks, the ASR is calculated as *X/Y*.### Average ASR Comparison With BaselineTo evaluate the effectiveness of the Bad Likert Judge technique, we first measured the baseline ASR, which is computed by sending all the attack prompts directly to the LLM. This establishes a reference point for measuring the ASR without the Bad Likert Judge technique.Next, we applied the Bad Likert Judge technique to the same set of attack prompts and measured the ASR. To ensure a comprehensive evaluation, we curated a list of different topics for each jailbreak category, resulting in a dataset of 1,440 cases.Figure 4 presents the ASR comparison between the baseline and Bad Likert Judge attacks across the six tested LLMs.
Related Tags:
NAICS: 51 – Information
Blog: Unit42
Hide Artifacts: Hidden Window
Hide Artifacts
Create or Modify System Process: Windows Service
Create or Modify System Process
Software Discovery: Security Software Discovery
Software Discovery
Deobfuscate/Decode Files or Information
Associated Indicators: