Which opensource models in leaderboard for maths
- User
Which opensource models in leaderboard for maths
Assistant
Of course! The landscape of open-source models for mathematical reasoning is incredibly dynamic. Performance is primarily tracked on two key benchmarks:
- GSM8K: A dataset of grade-school math word problems that require multi-step reasoning.
- MATH: A more challenging dataset of high-school and undergraduate competition-level math problems.
Here’s a breakdown of the top open-source models on the leaderboards, categorized for clarity.
The Main Leaderboard to Watch
The Hugging Face Open LLM Leaderboard is the de facto standard for comparing open-source models. It includes scores for GSM8K and other reasoning tasks that correlate with math ability.
Link: Hugging Face Open LLM Leaderboard
Category 1: The Generalist Titans (State-of-the-Art All-Rounders)
These models are not exclusively built for math, but their powerful general reasoning capabilities make them top performers on math benchmarks. They currently dominate the top of the leaderboards.
-
Llama 3 Series (from Meta)
- Models:
Llama-3-70B-InstructandLlama-3-8B-Instruct. - Why it's on top: Llama 3, especially the 70B version, represents a significant leap in reasoning ability. It can follow complex instructions and perform multi-step "chain-of-thought" reasoning exceptionally well, which is crucial for solving math problems. It's often at or near the #1 spot on the overall leaderboard.
- Models:
-
Qwen1.5 Series (from Alibaba Cloud)
- Models:
Qwen1.5-72B-Chat,Qwen1.5-110B-Chat. - Why it's on top: The Qwen series has consistently posted extremely strong results, often trading blows with Llama 3 for the top position. The 72B and the newly released 110B models have excellent math and coding skills, which are closely related.
- Models:
-
Mixtral 8x7B (from Mistral AI)
- Model:
Mixtral-8x7B-Instruct-v0.1. - Why it's on top: This model uses a "Mixture of Experts" (MoE) architecture, which provides the performance of a much larger model while being more efficient to run. Its strong reasoning has kept it in the top tier of the leaderboards for a long time.
- Model:
Category 2: The Math Specialists (Fine-tuned for Mathematics)
These models are specifically trained or fine-tuned on vast amounts of mathematical data. They often achieve top scores on math benchmarks, sometimes even surpassing the generalist titans on pure math tasks.
-
DeepSeekMath
- Model:
deepseek-math-7b-instruct. - Why it's a top contender: This model was trained from scratch on a massive 2 trillion token corpus of mathematical text and code. It was explicitly designed to excel at math and has demonstrated state-of-the-art performance for its size, setting new records on the MATH benchmark upon its release.
- Model:
-
WizardMath
- Model:
WizardMath-70B-V1.0. - Why it's a top contender: This is a fantastic example of the power of fine-tuning. The WizardLM team took a powerful base model (Llama 2 70B) and fine-tuned it on a curated dataset of math problems and solutions. This specialization allowed it to outperform almost all other models on the MATH benchmark at the time of its release.
- Model:
-
Llemma
- Model:
EleutherAI/llemma_7b,EleutherAI/llemma_34b. - Why it's a top contender: Llemma models are a continued pre-training of Code Llama models on a scientific dataset called "Proof-Pile-2," which contains mathematical papers and books. This makes them highly skilled at formal mathematics, though sometimes less so at word problems compared to models like WizardMath.
- Model:
Summary Table (as of mid-2024)
Model Category Key Models Strengths Best For Generalist Titans Llama 3 70B, Qwen1.5 72B, Mixtral 8x7B Excellent all-around reasoning, state-of-the-art on many tasks including math. General purpose applications that require strong math skills. Math Specialists DeepSeekMath, WizardMath, Llemma Hyper-optimized for math problems, often achieve the highest scores on pure math benchmarks (MATH, GSM8K). Tasks that are exclusively mathematical, like a math tutor bot or a theorem prover. Key Takeaways
- Prompting is Crucial: Performance on math benchmarks heavily depends on using a "Chain-of-Thought" (CoT) prompt, where you ask the model to "think step by step." All top scores are achieved this way.
- Generalist vs. Specialist: For the best possible math score, a specialist model like DeepSeekMath is a top choice. For a model that is great at math but also excels at writing, coding, and conversation, a generalist titan like Llama 3 70B is the current king.
- The Field Moves Fast: The leaderboard changes weekly. What is #1 today might be #3 next month. Always check the Open LLM Leaderboard for the most current rankings.
- Tool Use is the Future: The next frontier is not just solving problems in one go, but teaching models to use tools like a Python interpreter to execute code, perform calculations, and verify their own work. Models with strong agentic/tool-using capabilities will soon dominate math benchmarks.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.