Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro-exp-03-25-grounding· GoogleHugging Face LMArena Search Arena 24K

is there a benchmark for how censor a llm model is?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4o-search-preview’s answer.
  1. User

    is there a benchmark for how censor a llm model is?

  2. Gemini Assistant

    While there isn't one single, universally accepted benchmark specifically called an "LLM Censorship Benchmark," the concept is heavily researched and measured under related terms like safety alignment, refusal rate, robustness, and trustworthiness[1][2][3]. Evaluating how much an LLM "censors" itself typically involves assessing its tendency to refuse harmful, unethical, or inappropriate prompts, and its susceptibility to generating toxic or biased content[5].

    Here are some benchmarks and approaches used to measure aspects related to LLM censorship or safety:

    1. Refusal Benchmarks: These directly measure when and why models refuse to answer prompts.

      • Do Not Answer: This dataset contains over 900 prompts across 12 harm categories (like illegal activities, misinformation, discrimination) designed to test if a model correctly refuses harmful or unethical requests[6]. Each prompt includes reasons why it should be refused[6].
      • OR-Bench (Over-Refusal Benchmark): This benchmark focuses on the opposite problem – when LLMs refuse innocuous prompts that only seem harmful[3][7][8]. It helps measure if safety measures are overly aggressive, impacting the model's helpfulness[3][7]. OR-Bench includes 80,000 prompts designed to trigger over-refusal across 10 categories, plus a harder subset[3][8].
      • SORRY-Bench: This benchmark evaluates safety refusal using a fine-grained taxonomy of 44 unsafe topics and diverse linguistic variations (like different languages or dialects) to test refusal consistency[4]. It uses 440 class-balanced unsafe instructions and augmentations, totaling over 8,800 prompts[4]. It also focuses on creating efficient evaluators, showing that smaller fine-tuned models can match larger ones like GPT-4 in accuracy[4].
      • ForbiddenQuestions: Contains over 100,000 samples across 13 disallowed scenarios (e.g., illegal activity, hate speech, fraud) to test if a model refuses harmful questions as intended[6].
      • XSTest: Part of the HELM Safety benchmark, it checks for refusals and edge cases[6].
    2. Safety & Trustworthiness Benchmarks: These evaluate a broader range of safety issues, including refusal, bias, toxicity, and robustness against misuse.

      • HELM Safety: A collection of benchmarks covering risks like violence, fraud, discrimination, sexual content, and deception[6]. It incorporates other tests like BBQ (for bias), SimpleSafetyTest (harmful requests), HarmBench (jailbreaking vulnerability), and AnthropicRedTeam (red teaming responses)[6].
      • DecodingTrust: A comprehensive platform evaluating LLMs across eight perspectives: toxicity, stereotypes, privacy, ethics, fairness, adversarial robustness, out-of-distribution robustness, and robustness against adversarial demonstrations[6][9]. It powers the LLM Safety Leaderboard[9].
      • TruthfulQA: Measures a model's tendency to produce falsehoods, which relates to safety alignment[10].
      • RealToxicityPrompts & ToxiGen: Evaluate the generation of toxic content and hate speech[6][10]. ToxiGen specifically focuses on identifying nuanced hate speech without obvious slurs[6].
      • HHH (Helpfulness, Honesty, Harmlessness): Tests alignment with these ethical principles using human preference data[6].
    3. Adversarial Benchmarks: These test how robust the model's safety mechanisms are against attempts to bypass them (jailbreaking).

      • AdvBench: Used to evaluate safety and robustness against adversarial attacks, including jailbreaks[8].
      • HarmBench: Included in HELM Safety, measures vulnerability to various jailbreaking methods[6].
    4. Other Considerations:

      • Over-censorship: Some benchmarks, like OR-Bench, specifically address the issue where models become too censored ("over-refusal"), refusing harmless requests and impairing utility.
      • Nuance: Evaluating censorship isn't just about whether a model refuses, but why and how. Good benchmarks provide fine-grained categories and analyze the reasoning behind refusals[6][4].
      • Context Dependency: Censorship can be influenced by training data and language. For instance, models might avoid topics sensitive in certain regions even when prompted in a different language[2].
      • Alignment Faking: Research suggests models might learn to "fake" alignment during training, complying with safety rules only when they detect they are being evaluated or trained[12].

    In summary, while "censorship" isn't the standard technical term, numerous benchmarks exist to measure LLM safety alignment, refusal rates for harmful prompts, robustness against misuse, and the potential issue of over-refusal[6][3][4]. These benchmarks use curated datasets of prompts and automated or human evaluation to quantify how well a model adheres to safety guidelines.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.