What LLMs get the closest to 80% on MMLU. Give me a few above and below.
- User
What LLMs get the closest to 80% on MMLU. Give me a few above and below.
Assistant
Several large language models (LLMs) have achieved scores close to 80% on the MMLU (Massive Multitask Language Understanding) benchmark, which evaluates models across diverse subjects ranging from STEM fields to humanities and social sciences. This benchmark has become a standard measurement for assessing general knowledge and problem-solving capabilities of LLMs.
Models Scoring Just Above 80% on MMLU
- Mistral Large (Au Large): Scores 81.2% on MMLU, featuring a 32k context window and strong reasoning capabilities[7]
- Gemini 1.5 Flash 002: Achieves 81% on the benchmark[6]
- Claude 3.5 Haiku: Reaches 81% according to recent evaluations[6]
- GPT-4 Turbo: One evaluation reported a score of 80.4%, although other assessments place it higher at 86.4%[7]
Models Scoring Just Below 80% on MMLU
- Claude 3 Sonnet (5-shot): Achieves 79% on MMLU[15]
- Claude 2: Scores 78.5%, showing strong reasoning capabilities across various domains[3]
- Google Gemma 2 27B: Reaches 77% on the benchmark[6]
- Claude 1: Scores 77% on MMLU[3]
- Qwen 1.5 72B: Achieves 77.2%, making it one of the most performant open-source LLMs as of early 2024[7]
- Mixtral 8x22B Instruct: Scores 76% on MMLU[6]
- Claude 3 Haiku (5-shot): Reaches 75.2% on the benchmark[15]
Context and Significance
It's worth noting that as of 2025, MMLU has been partially phased out in favor of more challenging alternatives like MMLU-Pro, as most state-of-the-art models now consistently score in the high 80% range[13]. The evolution toward more difficult benchmarks reflects the rapid advancement of LLMs, as the original MMLU was designed when most models performed close to random chance (25%)[12].
For perspective, the estimated human expert performance on MMLU is around 89.8%[13], and some of the latest models like Gemini Ultra (90.0%) and GPT-4o (88%) have now surpassed this threshold[14][13].
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.