Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-flash-preview-04-17-grounding· GoogleHugging Face LMArena Search Arena 24K

can you make me a benchmarks comparison of o4-mini vs deepseek r1 performance benchmarks?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4o-mini-search-preview’s answer.
  1. User

    can you make me a benchmarks comparison of o4-mini vs deepseek r1 performance benchmarks?

  2. Gemini Assistant

    Here is a comparison of the performance benchmarks for OpenAI's o4-mini and DeepSeek R1 based on the available information:

    Model Overview

    • OpenAI o4-mini: Released in April 2025, o4-mini is a smaller model optimized for fast, cost-efficient reasoning[1][2]. It is designed to handle math, coding, and visual tasks efficiently[1]. It supports multimodal input (text and images) and integrates tool use capabilities[1][4][5].
    • DeepSeek R1: Released in January 2025, DeepSeek R1 is an open-source Mixture-of-Experts (MoE) model with 671 billion parameters (37B active per token). It was trained with a focus on reasoning capabilities using reinforcement learning[6][7][8]. DeepSeek R1 primarily supports text-based input.

    Benchmark Comparison

    Here's a comparison of their performance on various benchmarks:

    BenchmarkCategoryDeepSeek R1 Scoreo4-mini ScoreNotes
    MMLU-ProReasoning & Knowledge84%83% (o4-mini high)[10, 14]
    GPQA DiamondScientific Reasoning71.5%81.4%[1, 3, 6, 7]. o4-mini high: 78% [10].
    MATH-500Quantitative Reasoning97.3%N/ADeepSeek R1 leads OpenAI o1 [1, 6].
    AIME 2024Competition Math79.8%93.4% (no tools)o4-mini performs significantly better [3, 6]. With Python, o4-mini scores 99.5% [2][1].
    CodeforcesCoding2029 Elo2719 Elo (with terminal)o4-mini performs significantly better [3, 6].
    SWE-Bench VerifiedCoding49.2%68.1%o4-mini performs better [3, 6].
    LiveCodeBenchCodingVaries (distilled models ~50-57%)N/A[1] (Scores for DeepSeek distilled models). o4-mini high included in index [10][5].
    SciCodeCodingN/AIncluded in indexIncluded in Artificial Analysis Index [10].
    HumanEvalCodingN/AN/A[13] states N/A for DeepSeek R1. Scores for o4-mini not readily available.
    MMMUMultimodal ReasoningDoes not support images81.6%o4-mini supports multimodal reasoning [2, 3, 5, 14, 15].
    MathVistaVisual Math ReasoningDoes not support images84.3%o4-mini supports multimodal reasoning [2, 3, 5, 14, 15].
    CharXiv (Reasoning)Scientific Figure ReasoningDoes not support images72.0%o4-mini supports multimodal reasoning [2, 3, 5, 14, 15].

    Other Key Comparisons

    • Context Window:
      • o4-mini: 200,000 tokens [14, 15]
      • DeepSeek R1: 128,000 [14, 15] or 130,000 tokens [14]
      • o4-mini has a larger context window.
    • Speed (Output Tokens per Second):
      • o4-mini (high): 132 t/s [10]
      • DeepSeek R1: 24-25.3 t/s [10, 14]
      • o4-mini is significantly faster in terms of output speed.
    • Latency (Time to First Token):
      • o4-mini (high): 7.32s [6]
      • DeepSeek R1: 3.23s [14][10]
      • DeepSeek R1 appears to have lower latency, but these figures are from different sources and might not be directly comparable.
    • Price (USD per 1M Tokens):
      • o4-mini: 1.10(input),1.10 (input), 4.40 (output) [6]
      • DeepSeek R1: 0.55(input),0.55 (input), 2.19 (output) [14][10]
      • DeepSeek R1 is roughly 2.0x cheaper than o4-mini [14, 15].
    • Multimodal Capabilities:
      • o4-mini: Supports text and image inputs [14, 15][3][4][5].
      • DeepSeek R1: Primarily text-based input [14, 15].
      • o4-mini has a significant advantage in multimodal tasks[3][5].
    • Tool Use:
      • o4-mini: Explicitly supports tool use (web search, Python interpreter, etc.) [2, 3, 6].
      • DeepSeek R1: Training incorporates RL, but explicit external tool use for the end model is not as prominently mentioned as for o4-mini.
    • Open Source:
      • o4-mini: Closed-source (OpenAI).
      • DeepSeek R1: Open source [6, 7][6][7][11].

    Summary

    Based on the benchmarks and features:

    • o4-mini generally shows stronger performance on coding benchmarks (Codeforces, SWE-Bench), multimodal reasoning tasks, and competitive math (AIME). It also has a larger context window and significantly faster output speed. It supports native multimodal input and tool use[3][4][5].
    • DeepSeek R1 demonstrates very strong performance in quantitative reasoning (MATH-500) and is competitive in general reasoning/knowledge (MMLU-Pro, GPQA Diamond). Its key advantages are its significantly lower cost and its open-source nature.

    The choice between the two models would depend on the specific requirements of the task, prioritizing factors like cost, speed, multimodal needs, coding performance, and the importance of an open-source model.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.