what are some hard but notable LLM benchmarks? I mean those that are not yet saturated.…
- User
what are some hard but notable LLM benchmarks? I mean those that are not yet saturated. Please verify that the benchmarks are indeed not saturated as of 2025. For saturated I mean: LLMs do not score 80% of more of the possible points in the benchmark.
- Assistant
As of April 2025, several challenging and notable benchmarks for Large Language Models (LLMs) remain unsaturated, with top-performing models scoring below 80%. Here are some of these benchmarks:
-
Advanced Reasoning Benchmark (ARB): Introduced in July 2023, ARB comprises advanced reasoning problems across multiple fields, including mathematics, physics, biology, chemistry, and law. Evaluations of models like GPT-4 and Claude have shown scores well below 50% on more demanding tasks, indicating significant room for improvement[1].
-
LiveBench: Released in June 2024, LiveBench is designed to be contamination-free and challenging, featuring tasks from recent math competitions, arXiv papers, and news articles. It covers a wide array of tasks such as math, coding, reasoning, language, instruction following, and data analysis. Top models have achieved below 65% accuracy, highlighting its difficulty[2].
-
MultiChallenge: Published in January 2025, MultiChallenge evaluates LLMs on conducting multi-turn conversations with human users. It presents four categories of challenges that require accurate instruction-following, context allocation, and in-context reasoning. Despite near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet achieving just 41.4% average accuracy[3].
-
GPQA (Google-Proof Q&A): This benchmark consists of 448 multiple-choice questions in biology, physics, and chemistry, designed to be "Google-proof." Even skilled professionals with unlimited web access achieve only 34% accuracy. As of 2024, top models like Claude 3.5 Sonnet and GPT-4o have scores of 59.4% and 53.6%, respectively, indicating that the benchmark remains unsaturated[4].
-
MATH Benchmark: Containing 12,500 competition-level mathematics problems, the MATH benchmark assesses mathematical reasoning. As of 2024, leading models like GPT-4o and Meta Llama 3.1 405B have achieved scores of 76.6% and 73.8%, respectively, suggesting that the benchmark is approaching saturation but still presents challenges[4].
These benchmarks continue to provide rigorous assessments of LLM capabilities, highlighting areas where current models have significant room for improvement.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.