Skip to content
TheChatArchiveThe Chat Archive
Qwenqwen3-235b-a22b-no-thinking· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

I want to know the latest ranking of model programming capabilities.

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwen3-30b-a3b’s answer.
  1. User

    I want to know the latest ranking of model programming capabilities.

  2. Qwen Assistant

    As of now, there is no single official ranking for the programming capabilities of large language models (LLMs), but several organizations and research institutions have developed benchmarks and evaluations to compare the performance of different models. Below are some of the most recognized benchmarking platforms and recent findings up to 2024:


    📊 Top Benchmarks for Programming Capabilities

    1. HumanEval (by OpenAI)

      • Measures the model’s ability to generate correct Python code based on function descriptions.
      • Metrics: Pass@k (e.g., Pass@1 or Pass@10)
    2. MBPP (Mostly Basic Python Problems)

      • A set of Python coding problems intended for introductory-level programmers.
    3. Codeforces / Competitive Programming Evaluation

      • Some studies evaluate how well models perform on competitive programming tasks.
    4. APPS (A Benchmark for Code Generation with Applications in Program Synthesis)

      • Includes both introductory and competition-level programming problems.
    5. SWE-bench

      • Evaluates the model's ability to fix real-world GitHub issues.
    6. LiveCodeBench

      • Focuses on generating executable code in dynamic environments.
    7. MultiPL-E

      • Extends HumanEval into multiple programming languages.
    8. CodeGen (Salesforce) & CodeXGLUE

      • Also used in various academic comparisons.

    🏆 Top Performing Models in Programming (as of late 2024)

    ModelDeveloperHumanEval Pass@1Notes
    DeepSeek-Coder-V2DeepSeek~85%One of the top-performing open-source models in code generation.
    Qwen-Coder / Qwen-Coder-PlusAlibaba Cloud~80–83%Strong across multiple languages and code reasoning tasks.
    StarCoder2 seriesBigCode (Hugging Face)~70–75%Large training data from The Stack v2.
    CodeLlama seriesMeta~70–73%Based on Llama 2, available in multiple variants.
    GPT-4o / GPT-4 TurboOpenAI~90%+Proprietary, industry-leading in many code tasks.
    Claude 3 seriesAnthropic~85–88%Strong reasoning and code generation abilities.
    Gemini Pro 1.5Google~80–85%Good at multi-step reasoning and code synthesis.

    🚀 Note: GPT-4o, Claude 3, and Gemini Pro 1.5 tend to dominate due to their massive scale and proprietary training data, but models like DeepSeek-Coder-V2 and Qwen-Coder are catching up in open-weight categories.


    🔍 How to Test a Model’s Programming Skills Yourself


    🧠 Conclusion

    If you're looking for the best programming model, here’s a quick summary:

    Use CaseTop Choice
    Best Overall (Proprietary)Claude 3 Opus / GPT-4o
    Best Open-Weight ModelDeepSeek-Coder-V2 / Qwen-Coder-Plus
    Best for Code CompletionCodeLlama / StarCoder2
    Best for Real-World Bugs FixingSWE-agent / AgentCoder / LiveCodeBench-capable models

    Would you like me to help you test specific models or recommend one based on your use case (e.g., Python only, C++, or full-stack development)?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.