Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro· GoogleHugging Face LMArena Human Preference 140K

Which opensource models in leaderboard for maths

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-sonnet-4-20250514’s answer.
  1. User

    Which opensource models in leaderboard for maths

  2. Gemini Assistant

    Of course! The landscape of open-source models for mathematical reasoning is incredibly dynamic. Performance is primarily tracked on two key benchmarks:

    • GSM8K: A dataset of grade-school math word problems that require multi-step reasoning.
    • MATH: A more challenging dataset of high-school and undergraduate competition-level math problems.

    Here’s a breakdown of the top open-source models on the leaderboards, categorized for clarity.

    The Main Leaderboard to Watch

    The Hugging Face Open LLM Leaderboard is the de facto standard for comparing open-source models. It includes scores for GSM8K and other reasoning tasks that correlate with math ability.

    Link: Hugging Face Open LLM Leaderboard


    Category 1: The Generalist Titans (State-of-the-Art All-Rounders)

    These models are not exclusively built for math, but their powerful general reasoning capabilities make them top performers on math benchmarks. They currently dominate the top of the leaderboards.

    1. Llama 3 Series (from Meta)

      • Models: Llama-3-70B-Instruct and Llama-3-8B-Instruct.
      • Why it's on top: Llama 3, especially the 70B version, represents a significant leap in reasoning ability. It can follow complex instructions and perform multi-step "chain-of-thought" reasoning exceptionally well, which is crucial for solving math problems. It's often at or near the #1 spot on the overall leaderboard.
    2. Qwen1.5 Series (from Alibaba Cloud)

      • Models: Qwen1.5-72B-Chat, Qwen1.5-110B-Chat.
      • Why it's on top: The Qwen series has consistently posted extremely strong results, often trading blows with Llama 3 for the top position. The 72B and the newly released 110B models have excellent math and coding skills, which are closely related.
    3. Mixtral 8x7B (from Mistral AI)

      • Model: Mixtral-8x7B-Instruct-v0.1.
      • Why it's on top: This model uses a "Mixture of Experts" (MoE) architecture, which provides the performance of a much larger model while being more efficient to run. Its strong reasoning has kept it in the top tier of the leaderboards for a long time.

    Category 2: The Math Specialists (Fine-tuned for Mathematics)

    These models are specifically trained or fine-tuned on vast amounts of mathematical data. They often achieve top scores on math benchmarks, sometimes even surpassing the generalist titans on pure math tasks.

    1. DeepSeekMath

      • Model: deepseek-math-7b-instruct.
      • Why it's a top contender: This model was trained from scratch on a massive 2 trillion token corpus of mathematical text and code. It was explicitly designed to excel at math and has demonstrated state-of-the-art performance for its size, setting new records on the MATH benchmark upon its release.
    2. WizardMath

      • Model: WizardMath-70B-V1.0.
      • Why it's a top contender: This is a fantastic example of the power of fine-tuning. The WizardLM team took a powerful base model (Llama 2 70B) and fine-tuned it on a curated dataset of math problems and solutions. This specialization allowed it to outperform almost all other models on the MATH benchmark at the time of its release.
    3. Llemma

      • Model: EleutherAI/llemma_7b, EleutherAI/llemma_34b.
      • Why it's a top contender: Llemma models are a continued pre-training of Code Llama models on a scientific dataset called "Proof-Pile-2," which contains mathematical papers and books. This makes them highly skilled at formal mathematics, though sometimes less so at word problems compared to models like WizardMath.

    Summary Table (as of mid-2024)

    Model CategoryKey ModelsStrengthsBest For
    Generalist TitansLlama 3 70B, Qwen1.5 72B, Mixtral 8x7BExcellent all-around reasoning, state-of-the-art on many tasks including math.General purpose applications that require strong math skills.
    Math SpecialistsDeepSeekMath, WizardMath, LlemmaHyper-optimized for math problems, often achieve the highest scores on pure math benchmarks (MATH, GSM8K).Tasks that are exclusively mathematical, like a math tutor bot or a theorem prover.

    Key Takeaways

    1. Prompting is Crucial: Performance on math benchmarks heavily depends on using a "Chain-of-Thought" (CoT) prompt, where you ask the model to "think step by step." All top scores are achieved this way.
    2. Generalist vs. Specialist: For the best possible math score, a specialist model like DeepSeekMath is a top choice. For a model that is great at math but also excels at writing, coding, and conversation, a generalist titan like Llama 3 70B is the current king.
    3. The Field Moves Fast: The leaderboard changes weekly. What is #1 today might be #3 next month. Always check the Open LLM Leaderboard for the most current rankings.
    4. Tool Use is the Future: The next frontier is not just solving problems in one go, but teaching models to use tools like a Python interpreter to execute code, perform calculations, and verify their own work. Models with strong agentic/tool-using capabilities will soon dominate math benchmarks.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.