Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-reasoning-pro-high· PerplexityHugging Face LMArena Search Arena 24K

What LLMs get the closest to 80% on MMLU. Give me a few above and below.

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-pro-high’s answer.
  1. User

    What LLMs get the closest to 80% on MMLU. Give me a few above and below.

  2. Perplexity Assistant

    Several large language models (LLMs) have achieved scores close to 80% on the MMLU (Massive Multitask Language Understanding) benchmark, which evaluates models across diverse subjects ranging from STEM fields to humanities and social sciences. This benchmark has become a standard measurement for assessing general knowledge and problem-solving capabilities of LLMs.

    Models Scoring Just Above 80% on MMLU

    • Mistral Large (Au Large): Scores 81.2% on MMLU, featuring a 32k context window and strong reasoning capabilities[7]
    • Gemini 1.5 Flash 002: Achieves 81% on the benchmark[6]
    • Claude 3.5 Haiku: Reaches 81% according to recent evaluations[6]
    • GPT-4 Turbo: One evaluation reported a score of 80.4%, although other assessments place it higher at 86.4%[7]

    Models Scoring Just Below 80% on MMLU

    • Claude 3 Sonnet (5-shot): Achieves 79% on MMLU[15]
    • Claude 2: Scores 78.5%, showing strong reasoning capabilities across various domains[3]
    • Google Gemma 2 27B: Reaches 77% on the benchmark[6]
    • Claude 1: Scores 77% on MMLU[3]
    • Qwen 1.5 72B: Achieves 77.2%, making it one of the most performant open-source LLMs as of early 2024[7]
    • Mixtral 8x22B Instruct: Scores 76% on MMLU[6]
    • Claude 3 Haiku (5-shot): Reaches 75.2% on the benchmark[15]

    Context and Significance

    It's worth noting that as of 2025, MMLU has been partially phased out in favor of more challenging alternatives like MMLU-Pro, as most state-of-the-art models now consistently score in the high 80% range[13]. The evolution toward more difficult benchmarks reflects the rapid advancement of LLMs, as the original MMLU was designed when most models performed close to random chance (25%)[12].

    For perspective, the estimated human expert performance on MMLU is around 89.8%[13], and some of the latest models like Gemini Ultra (90.0%) and GPT-4o (88%) have now surpassed this threshold[14][13].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.