Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-pro· PerplexityHugging Face LMArena Search Arena 24K

Give me a breakdown comparing all benchmarks as accurately as possible between the newly…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4o-search-preview-high-loc’s answer.
  1. User

    Give me a breakdown comparing all benchmarks as accurately as possible between the newly released o series of models from openai and Google best gemini models till date

  2. Perplexity Assistant

    OpenAI o-Series vs. Google Gemini: Benchmark Comparison

    This breakdown compares the latest OpenAI o-series models (notably o1, o3, and o4-mini) with Google's best Gemini models (Ultra, Pro, and 1.5) across key academic, reasoning, coding, and multimodal benchmarks. The focus is on the most recent and capable models as of April 2025.


    Model Lineup Overview

    CompanyModel SeriesNotable VariantsRelease Notes
    OpenAIo-serieso1, o3, o4-minio3 is the current flagship for reasoning and general intelligence; o4-mini is optimized for speed and cost-efficiency[3][4].
    GoogleGeminiUltra, Pro, 1.5Ultra is the top-tier, Pro is widely deployed, 1.5 is the latest major update[5][8].

    Benchmark Performance Comparison

    1. Reasoning & General Intelligence

    BenchmarkOpenAI o3 / o1 / o4-miniGoogle Gemini Ultra / Pro / 1.5
    ARC-AGIo3: 75.7% (std), 87.5% (high compute) – a major leap in adaptability and generalization[1].No public Gemini Ultra result; previous Gemini models did not match this level.
    MMLUo1: 90.8[7]; o3: SOTA (exact % not public, but outperforms o1)[4]; o4-mini: not specified.Gemini Pro 1.5: 81.9; Gemini Flash 1.5: 78.9; Gemini Ultra: claimed SOTA in 2023, but recent results lag behind o-series[5][8].
    GPQAo1: 73.3 (graduate-level reasoning)[7]; o3: higher, but not specified.Gemini Pro 1.5: 41.5; Gemini Ultra: not specified[8].
    MMMU (Vision Reasoning)o1: 78.2% (first to match human experts)[2]; o3: SOTA[4].Gemini Ultra: SOTA in 2023, but recent head-to-heads favor o-series[5].

    2. Math & STEM

    BenchmarkOpenAI o-seriesGoogle Gemini
    AIME 2024o1: 74% (single), 83% (consensus), 93% (reranked)[2]; o4-mini: best-performing model on AIME 2024/2025[4].Gemini 1.5: 58.5[8].
    MATHo1-preview: 85.5[7]; o4-mini: SOTA[4].Gemini 2: 89.7[7]; Gemini Pro 1.5: 58.5[8].
    Codeforceso3: SOTA[4]; o1-preview: 1258[7].Gemini: No public data[7].

    3. Coding

    BenchmarkOpenAI o-seriesGoogle Gemini
    HumanEvalo1-preview: 92.4[7]; o3: SOTA[4].Gemini 2: 92.9[7]; Gemini Pro 1.5: 71.9[8].
    SWE-bencho3: SOTA (no custom scaffolding)[4].No public Gemini result.

    4. Multimodal & Translation

    BenchmarkOpenAI o-seriesGoogle Gemini
    Vision Taskso3: SOTA on image, chart, and graphic analysis[4].Gemini Ultra: SOTA in 2023, but recent head-to-heads favor o-series[5].
    Translation (WMT23)Not specified.Gemini Ultra: 74.4 BLEURT (all langs), slightly ahead of GPT-4 (73.8)[5].

    Key Strengths and Weaknesses

    OpenAI o-Series:

    • o3 and o4-mini set new state-of-the-art (SOTA) scores on reasoning, coding, and math benchmarks, with o3 showing a major leap in generalization and adaptability (ARC-AGI)[1][4].
    • o1 and o3 outperform Gemini models on MMLU, GPQA, and coding tasks[7][8].
    • o4-mini is highly efficient, with top math and coding performance for its size/cost[4].
    • o3 demonstrates 20% fewer major errors than o1 on real-world tasks[4].

    Google Gemini:

    • Gemini Ultra was SOTA in 2023, especially in multimodal and translation tasks[5].
    • Gemini 2 matches or slightly exceeds o1-preview in some math/coding tasks (e.g., HumanEval, MATH), but lags in general reasoning and knowledge (MMLU, GPQA)[7][8].
    • Gemini Ultra leads in translation (WMT23) and remains strong in multilingual and multimodal benchmarks[5].

    Summary Table: Head-to-Head Scores

    BenchmarkOpenAI o3/o1/o4-miniGemini Ultra/Pro/1.5
    ARC-AGI75.7% (std), 87.5% (high compute)[1]Not available
    MMLU90.8 (o1)[7]81.9 (Pro 1.5)[8]
    GPQA73.3 (o1)[7]41.5 (Pro 1.5)[8]
    AIME 202474–93 (o1)[2]58.5 (Pro 1.5)[8]
    MATH85.5 (o1-preview)[7]89.7 (Gemini 2)[7]
    HumanEval92.4 (o1-preview)[7]92.9 (Gemini 2)[7]
    CodeforcesSOTA (o3)[4]Not available
    MMMU (Vision)78.2 (o1)[2]SOTA (Ultra, 2023)[5]
    WMT23 (BLEURT)Not available74.4 (Ultra)[5]

    Conclusion

    • OpenAI's o-series (especially o3 and o4-mini) now lead on most reasoning, coding, and general intelligence benchmarks, marking a significant leap over both previous GPT models and Google's Gemini line in these domains[1][4][7][8].
    • Google's Gemini Ultra remains highly competitive in multimodal and translation tasks, and still leads in some multilingual benchmarks[5].
    • For math and coding, both ecosystems are close, with Gemini 2 slightly ahead in MATH and HumanEval, but OpenAI o-series dominating in broader reasoning and knowledge tasks.

    Overall, OpenAI's o-series models have established a new state-of-the-art in reasoning and general intelligence, while Gemini remains strong in multimodal and translation capabilities.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.