Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-search-preview-high-loc· OpenAIHugging Face LMArena Search Arena 24K

what are some hard but notable LLM benchmarks? I mean those that are not yet saturated.…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.0-flash-grounding’s answer.
  1. User

    what are some hard but notable LLM benchmarks? I mean those that are not yet saturated. Please verify that the benchmarks are indeed not saturated as of 2025. For saturated I mean: LLMs do not score 80% of more of the possible points in the benchmark.

  2. Assistant

    As of April 2025, several challenging and notable benchmarks for Large Language Models (LLMs) remain unsaturated, with top-performing models scoring below 80%. Here are some of these benchmarks:

    1. Advanced Reasoning Benchmark (ARB): Introduced in July 2023, ARB comprises advanced reasoning problems across multiple fields, including mathematics, physics, biology, chemistry, and law. Evaluations of models like GPT-4 and Claude have shown scores well below 50% on more demanding tasks, indicating significant room for improvement[1].

    2. LiveBench: Released in June 2024, LiveBench is designed to be contamination-free and challenging, featuring tasks from recent math competitions, arXiv papers, and news articles. It covers a wide array of tasks such as math, coding, reasoning, language, instruction following, and data analysis. Top models have achieved below 65% accuracy, highlighting its difficulty[2].

    3. MultiChallenge: Published in January 2025, MultiChallenge evaluates LLMs on conducting multi-turn conversations with human users. It presents four categories of challenges that require accurate instruction-following, context allocation, and in-context reasoning. Despite near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet achieving just 41.4% average accuracy[3].

    4. GPQA (Google-Proof Q&A): This benchmark consists of 448 multiple-choice questions in biology, physics, and chemistry, designed to be "Google-proof." Even skilled professionals with unlimited web access achieve only 34% accuracy. As of 2024, top models like Claude 3.5 Sonnet and GPT-4o have scores of 59.4% and 53.6%, respectively, indicating that the benchmark remains unsaturated[4].

    5. MATH Benchmark: Containing 12,500 competition-level mathematics problems, the MATH benchmark assesses mathematical reasoning. As of 2024, leading models like GPT-4o and Meta Llama 3.1 405B have achieved scores of 76.6% and 73.8%, respectively, suggesting that the benchmark is approaching saturation but still presents challenges[4].

    These benchmarks continue to provide rigorous assessments of LLM capabilities, highlighting areas where current models have significant room for improvement.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.