Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-pro-high· PerplexityHugging Face LMArena Search Arena 24K

Give me a breakdown comparing all benchmarks as accurately as possible between the newly…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4o-search-preview-high’s answer.
  1. User

    Give me a breakdown comparing all benchmarks as accurately as possible between the newly released o series of models from openai and Google best gemini models till date

  2. Perplexity Assistant

    OpenAI o-Series vs. Google Gemini: Benchmark Comparison (April 2025)

    This breakdown compares OpenAI’s latest o-series models (o3, o3-mini, o4-mini, o1) with Google’s best Gemini models (Gemini Ultra, Gemini Pro, Gemini 2.5 Pro), focusing on the most recent and authoritative benchmark results across reasoning, coding, math, multimodal, and context capabilities.


    Model Overview

    ModelRelease DateContext WindowMultimodalNotable Features
    OpenAI o3Apr 2025200,000YesSOTA reasoning, vision, tool use
    OpenAI o3-miniJan 2025200,000YesEfficient, high accuracy
    OpenAI o4-miniApr 2025200,000YesFast, cost-efficient, top math
    OpenAI o1Dec 2024200,000YesStrong reasoning, vision
    Gemini UltraDec 202332,800YesSOTA multimodal, academic leader
    Gemini ProDec 202332,800YesCost/performance optimized
    Gemini 2.5 ProJan 20251,000,000+YesLatest, best Gemini reasoning

    Benchmark Table: Head-to-Head

    BenchmarkOpenAI o3OpenAI o3-miniOpenAI o4-miniOpenAI o1Gemini UltraGemini ProGemini 2.5 Pro
    MMLU (5-shot)90.8%*86.9%N/A90.8%*83.7%71.8%89.8% (Global)
    MMMU82.9%N/AN/A77.6%59.4%47.9%81.7%
    AIME 202491.6%N/A92%74.3%N/AN/A92%
    AIME 202588.9%N/A86.7%N/AN/AN/A86.7%
    HumanEval97%97%N/A92.4%74.4%67.7%N/A
    MATH97.9%97.9%N/A85.5%53.2%32.6%N/A
    GPQA Diamond83.3%N/AN/A78%N/AN/A84%
    ARC-AGI75.7% (std)N/AN/A5% (GPT-4o)N/AN/AN/A
    Codeforces2706N/AN/A1258Top 15% (AlphaCode 2)N/AN/A
    SWE-Bench69.1%N/A63.8%48.9%N/AN/A63.8%

    *o3 and o1 have similar MMLU scores, but o3 is superior in reasoning-heavy and real-world tasks.


    Key Benchmark Insights

    Reasoning & General Knowledge (MMLU, ARC-AGI, GPQA)

    • OpenAI o3 sets a new state-of-the-art (SOTA) on MMLU (90.8%) and ARC-AGI (75.7% standard compute, 87.5% high compute), far surpassing Gemini Ultra (MMLU 83.7%) and Gemini Pro (71.8%) in reasoning and generalization[1][5][13][16][17][18].
    • Gemini 2.5 Pro closes the gap on MMLU (89.8% Global), but ARC-AGI results are not available for Gemini[12].

    Math & Science (AIME, MATH, GPQA)

    • OpenAI o3/o4-mini dominate math benchmarks: AIME 2024 (91.6%), AIME 2025 (88.9%), MATH (97.9%), and GPQA Diamond (83.3%), outperforming Gemini Ultra and Pro by large margins[5][12][13][14].
    • Gemini 2.5 Pro matches o4-mini on AIME 2024 (92%) and AIME 2025 (86.7%), but lags on MATH and GPQA[12].

    Coding (HumanEval, Codeforces, SWE-Bench)

    • OpenAI o3/o3-mini achieve 97% on HumanEval and 2706 on Codeforces, compared to Gemini Ultra’s 74.4% and Gemini Pro’s 67.7%[5][13][14][16].
    • SWE-Bench: o3 leads with 69.1%, Gemini 2.5 Pro at 63.8% (with custom agent setup)[5][12].

    Multimodal & Visual Reasoning (MMMU, MathVista, CharXiv)

    • OpenAI o3: 82.9% on MMMU, 86.8% on MathVista, 78.6% on CharXiv-Reasoning, all well above Gemini Ultra’s 59.4% on MMMU[5][13][16][17].
    • Gemini 2.5 Pro: 81.7% on MMMU, nearly matching o3[12].

    Context Window

    • OpenAI o-series: 200,000 tokens (o3, o3-mini, o4-mini, o1)[3][13][14].
    • Gemini Ultra/Pro: 32,800 tokens[13][14][16].
    • Gemini 2.5 Pro: 1,000,000+ tokens, best for ultra-long context[12][15].

    Technical and Qualitative Differences

    OpenAI o-Series

    • SOTA in reasoning, math, coding, and visual tasks.
    • ARC-AGI performance marks a leap in generalization and adaptability.
    • Autonomous tool use and advanced vision integration.
    • Larger context window (200K tokens), efficient tool use, and structured outputs.
    • o4-mini offers top math/coding performance at lower cost and higher throughput[3][4][5].

    Google Gemini

    • Gemini Ultra: SOTA in multimodal benchmarks at launch, but now surpassed by o3 in most reasoning/coding/math tasks.
    • Gemini 2.5 Pro: Major improvements, especially in context length (1M+ tokens) and global MMLU, but still trails o3 in coding and math[12].
    • AlphaCode 2 (Gemini-powered) ranks top 15% on Codeforces, but o3’s ELO is higher[6][11].
    • Gemini excels in multilinguality and real-time web access, with strong performance in creative and research-based tasks[10][12].

    Summary Table: OpenAI o3 vs. Gemini Ultra/2.5 Pro

    AreaOpenAI o3/o4-miniGemini Ultra/2.5 ProWinner
    Reasoning (MMLU)90.8% (o3)83.7% (Ultra), 89.8% (2.5 Pro)o3/2.5 Pro
    Math (AIME/MATH)91.6%/97.9% (o3/o4-mini)92%/N/A (2.5 Pro), 53.2% (Ultra)o3/o4-mini
    Coding (HumanEval)97% (o3/o3-mini)74.4% (Ultra)o3/o3-mini
    Multimodal (MMMU)82.9% (o3)59.4% (Ultra), 81.7% (2.5 Pro)o3/2.5 Pro
    ARC-AGI75.7% (std), 87.5% (high)N/Ao3
    Context Window200K tokens32.8K (Ultra), 1M+ (2.5 Pro)Gemini 2.5 Pro
    Tool UseYes (autonomous)Yes (AlphaCode 2, etc.)Both

    Conclusion

    • OpenAI o3 and o4-mini are the new leaders in reasoning, math, coding, and visual benchmarks, with o3 setting new records on ARC-AGI and other challenging tasks.
    • Gemini 2.5 Pro narrows the gap in some areas (context window, global MMLU, AIME), but still trails in coding and math.
    • Gemini Ultra/Pro remain strong in multimodal and multilingual tasks but are now surpassed by OpenAI’s o-series in most core benchmarks.
    • For ultra-long context (1M+ tokens), Gemini 2.5 Pro is unmatched, but for accuracy and reasoning, OpenAI o3/o4-mini are currently the top performers.

    In summary: OpenAI’s o-series, especially o3 and o4-mini, are the new state-of-the-art for most academic, reasoning, coding, and visual benchmarks, while Gemini 2.5 Pro leads in context length and remains competitive in some multimodal and multilingual domains[5][12][13][14][17][18].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.