can you make me a benchmarks comparison of o4-mini vs deepseek r1 performance benchmarks?
- User
can you make me a benchmarks comparison of o4-mini vs deepseek r1 performance benchmarks?
Assistant
Here is a comparison of the performance benchmarks for OpenAI's o4-mini and DeepSeek R1 based on the available information:
Model Overview
- OpenAI o4-mini: Released in April 2025, o4-mini is a smaller model optimized for fast, cost-efficient reasoning[1][2]. It is designed to handle math, coding, and visual tasks efficiently[1]. It supports multimodal input (text and images) and integrates tool use capabilities[1][4][5].
- DeepSeek R1: Released in January 2025, DeepSeek R1 is an open-source Mixture-of-Experts (MoE) model with 671 billion parameters (37B active per token). It was trained with a focus on reasoning capabilities using reinforcement learning[6][7][8]. DeepSeek R1 primarily supports text-based input.
Benchmark Comparison
Here's a comparison of their performance on various benchmarks:
Benchmark Category DeepSeek R1 Score o4-mini Score Notes MMLU-Pro Reasoning & Knowledge 84% 83% (o4-mini high) [10, 14] GPQA Diamond Scientific Reasoning 71.5% 81.4% [1, 3, 6, 7]. o4-mini high: 78% [10]. MATH-500 Quantitative Reasoning 97.3% N/A DeepSeek R1 leads OpenAI o1 [1, 6]. AIME 2024 Competition Math 79.8% 93.4% (no tools) o4-mini performs significantly better [3, 6]. With Python, o4-mini scores 99.5% [2][1]. Codeforces Coding 2029 Elo 2719 Elo (with terminal) o4-mini performs significantly better [3, 6]. SWE-Bench Verified Coding 49.2% 68.1% o4-mini performs better [3, 6]. LiveCodeBench Coding Varies (distilled models ~50-57%) N/A [1] (Scores for DeepSeek distilled models). o4-mini high included in index [10][5]. SciCode Coding N/A Included in index Included in Artificial Analysis Index [10]. HumanEval Coding N/A N/A [13] states N/A for DeepSeek R1. Scores for o4-mini not readily available. MMMU Multimodal Reasoning Does not support images 81.6% o4-mini supports multimodal reasoning [2, 3, 5, 14, 15]. MathVista Visual Math Reasoning Does not support images 84.3% o4-mini supports multimodal reasoning [2, 3, 5, 14, 15]. CharXiv (Reasoning) Scientific Figure Reasoning Does not support images 72.0% o4-mini supports multimodal reasoning [2, 3, 5, 14, 15]. Other Key Comparisons
- Context Window:
- o4-mini: 200,000 tokens [14, 15]
- DeepSeek R1: 128,000 [14, 15] or 130,000 tokens [14]
- o4-mini has a larger context window.
- Speed (Output Tokens per Second):
- o4-mini (high): 132 t/s [10]
- DeepSeek R1: 24-25.3 t/s [10, 14]
- o4-mini is significantly faster in terms of output speed.
- Latency (Time to First Token):
- o4-mini (high): 7.32s [6]
- DeepSeek R1: 3.23s [14][10]
- DeepSeek R1 appears to have lower latency, but these figures are from different sources and might not be directly comparable.
- Price (USD per 1M Tokens):
- o4-mini: 4.40 (output) [6]
- DeepSeek R1: 2.19 (output) [14][10]
- DeepSeek R1 is roughly 2.0x cheaper than o4-mini [14, 15].
- Multimodal Capabilities:
- o4-mini: Supports text and image inputs [14, 15][3][4][5].
- DeepSeek R1: Primarily text-based input [14, 15].
- o4-mini has a significant advantage in multimodal tasks[3][5].
- Tool Use:
- o4-mini: Explicitly supports tool use (web search, Python interpreter, etc.) [2, 3, 6].
- DeepSeek R1: Training incorporates RL, but explicit external tool use for the end model is not as prominently mentioned as for o4-mini.
- Open Source:
- o4-mini: Closed-source (OpenAI).
- DeepSeek R1: Open source [6, 7][6][7][11].
Summary
Based on the benchmarks and features:
- o4-mini generally shows stronger performance on coding benchmarks (Codeforces, SWE-Bench), multimodal reasoning tasks, and competitive math (AIME). It also has a larger context window and significantly faster output speed. It supports native multimodal input and tool use[3][4][5].
- DeepSeek R1 demonstrates very strong performance in quantitative reasoning (MATH-500) and is competitive in general reasoning/knowledge (MMLU-Pro, GPQA Diamond). Its key advantages are its significantly lower cost and its open-source nature.
The choice between the two models would depend on the specific requirements of the task, prioritizing factors like cost, speed, multimodal needs, coding performance, and the importance of an open-source model.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.