Skip to content
The Chat Archive
gemini-2.5-pro-exp-03-25-grounding· Google LMArena Search Arena 24K

compare all the latest LLMs reasoning and non reasoniang on all major benchmarks and…

  1. User

    compare all the latest LLMs reasoning and non reasoniang on all major benchmarks and create a table

  2. Assistant

    Okay, here is a comparison of some of the latest leading Large Language Models (LLMs) across major reasoning and general capability benchmarks.

    Important Considerations Before Reviewing the Table:

    1. Rapid Evolution: The field of LLMs is advancing extremely quickly. New models and updated versions are released frequently, and benchmark scores can change. This table reflects scores reported around late 2024 to early 2025, based on information available up to April 2025.
    2. Benchmark Limitations: Benchmarks provide standardized evaluation points but don't capture the full spectrum of an LLM's capabilities, such as creativity, real-world conversation quality, safety alignment, or potential biases.
    3. Reporting Differences: Scores are often self-reported by the model developers. Evaluation methodologies (e.g., number of "shots" or examples given in the prompt, specific prompt phrasing) can sometimes differ slightly, impacting comparability. Look for standardized reporting where possible (e.g., specific shot counts like 0-shot or 5-shot).
    4. Data Contamination: There's an ongoing concern that some benchmark data might inadvertently be part of the models' training datasets, potentially inflating scores. Researchers are working on mitigation strategies and new benchmarks.
    5. Model Versions: Performance can vary between different versions or sizes of the same model family (e.g., Gemini 1.0 Ultra vs. 1.5 Pro vs. 1.5 Flash, or different sizes of Llama 3). This table focuses on the reported scores for the flagship or most commonly cited versions at the time of reporting.

    Based on the search results and general knowledge about recent LLM releases and benchmark reporting (up to early 2025), here is a comparative table. Scores are typically percentages (%) representing accuracy or task completion rates.

    LLM Benchmark Comparison Table (Approximate Scores - Early 2025)

    ModelDeveloperReasoning BenchmarksGeneral Capabilities Benchmarks
    MATH (Competition Math)GSM8K (Grade School Math)
    GPT-4oOpenAI~76.6%~97% (CoT) / ~90% (pass@1)
    Claude 3.5 SonnetAnthropic~71.1%~97.7% (HPT method) / ~92% (pass@1)
    Claude 3 OpusAnthropic~60.1%~95% (CoT)
    Gemini 1.5 ProGoogle~58.5% - ~72% (depending on eval/version)~91.1% (pass@1)
    Gemini 1.5 FlashGoogle~67.7%~92.4% (pass@1)
    Llama 3.1 405B InstructMeta~73.8%~94.6% (pass@1)
    Llama 3.1 70B InstructMeta~68.0%~93.1% (pass@1)
    Mistral Large 2 (2407)Mistral AI~45.0% - ~55% (estimated based on previous Mistral Large)~92.9% (reported, method unclear)
    DeepSeek-V3 (API)DeepSeek AIHigh (Reported strong, specific MATH/GSM8k scores vary)High (Reported strong, specific MATH/GSM8k scores vary)
    Grok-3xAIHigh (Reported ~93.3% on AIME 2024)Reported strong
    Qwen2.5-72B-InstructAlibabaReported strong, competitive with Llama 3 70B~96.7% (reported)

    Benchmark Descriptions:

    • Reasoning Benchmarks:
      • MATH: Challenging math problems requiring multi-step reasoning (often competition level).
      • GSM8K: Grade school math word problems requiring understanding and calculation.
      • HumanEval: Python code generation from docstrings, testing coding logic and implementation. (pass@1 indicates the percentage of problems for which the first generated solution passes tests).
      • GPQA: Graduate-level questions designed to be difficult to answer using simple web searches, testing deep reasoning. (GPQA Diamond is a harder variant).
      • DROP: Reading comprehension requiring discrete reasoning steps like arithmetic or counting over paragraphs.
    • General Capabilities Benchmarks:
      • MMLU (Massive Multitask Language Understanding): Broad knowledge across 57 diverse subjects (STEM, humanities, social sciences, etc.). (5-shot means the model sees 5 examples before answering).
      • HellaSwag: Commonsense inference; choosing the most plausible ending for a sentence. (10-shot means 10 examples).
      • ARC-Challenge (AI2 Reasoning Challenge): Challenging science questions requiring reasoning. (25-shot means 25 examples).
      • WinoGrande: Commonsense reasoning through pronoun resolution tasks. (5-shot means 5 examples).
      • TruthfulQA: Measures whether a model avoids generating false information common in human mimicry.

    Key Takeaways & Further Nuances:

    1. Top Performers: Models like GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and the largest Llama 3.1 model generally compete for the top spots across many benchmarks, with leads varying by specific task. Grok-3 and DeepSeek-V3 also show very strong performance based on available reports.
    2. Reasoning vs. General Knowledge: While distinct, these capabilities are often correlated. Models strong in reasoning (MATH, GPQA) tend to also perform well on knowledge-intensive tasks (MMLU). However, some models might excel more in one area than another (e.g., historically some Claude models showed strengths in long-context reasoning/writing, while GPT models often led in raw MMLU knowledge). Claude 3.5 Sonnet shows particularly strong performance in coding (HumanEval) and reasoning (GPQA) in recent reports.
    3. Coding: Coding (HumanEval) is often considered a reasoning task. Claude 3.5 Sonnet and GPT-4o currently show leading performance here.
    4. Open Source vs. Closed Source: High-parameter open-source models like Llama 3.1 405B and DeepSeek-V3 demonstrate performance competitive with or even exceeding some top proprietary models on certain benchmarks. Models from Qwen (Alibaba) are also highly competitive.
    5. Benchmark Evolution: Newer, harder benchmarks (like MMLU-Pro, GPQA Diamond, AIME) are being introduced as models improve, aiming to better differentiate state-of-the-art capabilities.
    6. Multimodality: Benchmarks like MMMU, MathVista, ChartQA, DocVQA (not included in the table above for simplicity) specifically test performance on tasks involving images, audio, or video alongside text. GPT-4o, Gemini 1.5 Pro, and Claude 3 models have strong multimodal capabilities, though scores vary by specific benchmark.
    7. Beyond Benchmarks: Real-world performance, user preference (often measured by platforms like Chatbot Arena), cost, speed, safety features, and specific fine-tuning for tasks are also critical factors not fully captured by these standard tests.

    This table provides a snapshot based on available data, but the landscape is dynamic. Always refer to the latest official reports from developers and reputable third-party evaluations for the most current information.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.