Skip to content
TheChatArchiveThe Chat Archive
Qwenqwq-32b· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

I mean, which is the current position of Mistral-7B-Instruct-v0.1 at the leaderboard of…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-flash-preview-04-17’s answer.
  1. User

    I mean, which is the current position of Mistral-7B-Instruct-v0.1 at the leaderboard of LLMs?

  2. Qwen Assistant

    As of July 2024, Mistral-7B-Instruct-v0.1 holds a strong position among open-source LLMs, particularly for its size (7 billion parameters). Below is a breakdown of its performance relative to other models on key benchmarks and leaderboards:


    Key Benchmark Performances (v0.1)

    BenchmarkMistral-7B-Instruct-v0.1Top Competitors (for context)
    MMLU~60-65% accuracyLlama 2 13B (~50-60%), Phi-2 7B (~60-65%), Qwen1 (~65-70%)
    TruthfulQA~60-65%Qwen1 (~70-75%), Llama 2 70B (~70%)
    ARC-C~70-75%Llama 2 13B (~60-65%), Phi-2 7B (~70%)
    **HumanEval (coding)~35% passedPhi-2 7B (~40-45%), Qwen1 (~45%)
    HellaSwag~80-85%Llama 2 13B (~75-80%), Qwen1 (~80-85%)
    BigBenchStrong for its sizeLlama 2 70B (~10-15% better on complex tasks)

    Leaderboard Position

    1. Among 7B Models:

      • Top performer: Often outperforms other 7B models like Qwen1, Phi-2 7B, and Llama 2 7B on MMLU, ARC, and HellaSwag.
      • Close competition: Phi-2 7B and Qwen1 sometimes edge ahead in specific tasks (e.g., coding, factual accuracy).
    2. Versus Larger Models:

      • Beats Llama 2 13B: On most benchmarks, Mistral-7B-v0.1 matches or exceeds the performance of the Llama 2 13B (a 13B parameter model).
      • Approaches Llama 2 70B: In some areas (e.g., reasoning and code generation), it reaches ~70-80% of the performance of the Llama 2 70B (a 70B model), which is remarkable for its size.
    3. Specialized Tasks:

      • Coding: Moderately strong (HumanEval ~35% pass rate), but lags behind Phi-2 7B (~40%) and Qwen1 (~45%).
      • Long Context: Its Sliding Window Attention (SWA) allows 32k token context, outperforming most 7B models (e.g., Llama 2 7B maxes at 4k tokens).
    4. Instruction Following:

      • The Instruct-tuned version is highly refined for chat-like interactions, often rivaling models like Qwen1 and Phi-2 in conversational tasks.

    Key Advantages

    • Efficient Architecture: Grouped-query attention (GQA) and SWA enable faster inference and longer context.
    • Apache 2.0 License: Commercially usable, unlike models with restrictive licenses (e.g., Llama 2’s limited commercial terms).
    • Small but Mighty: Outperforms many larger open-source models in cost-sensitive scenarios (e.g., local deployment).

    Areas for Improvement

    • Coding: Lags slightly behind Phi-2 7B and Qwen1.
    • Factual Accuracy: Slightly behind Qwen1 on TruthfulQA.
    • Latest Competitors: Newer models like Phi-2 7B v2 or Qwen1.5 might have surpassed it in specific tasks.

    Comparison to Closed-Source Models

    • GPT-3.5 / GPT-4: Significantly outperform Mistral-7B in overall capability, but these are closed-source and require API access.
    • Google Gemini (small versions): Gemmini (~7B) is a strong competitor, but Mistral-7B often edges ahead in open-source comparisons.

    Current Use Cases

    • Local Deployment: Ideal for running on consumer hardware (e.g., RTX 4090) via tools like Ollama or LM Studio.
    • Chatbots: Strong conversational performance due to instruction-tuning.
    • Research: A go-to model for open-source studies due to its balance of performance and license.

    Conclusion

    Mistral-7B-Instruct-v0.1 is among the top open-source LLMs for its size, consistently outperforming 13B models like Llama 2 and holding its own against other 7B competitors. While newer variants (e.g., Mistral 8x7B) and specialized models (e.g., Phi-2, Qwen) may edge ahead in specific niches, v0.1 remains a gold-standard baseline for open-source LLMs. Its Apache 2.0 license and efficient architecture make it a standout choice for developers and researchers.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.