Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-mini-search-preview· OpenAIHugging Face LMArena Search Arena 24K

what is SWE-Bench state of the art at the moment?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-reasoning’s answer.
  1. User

    what is SWE-Bench state of the art at the moment?

  2. Assistant

    As of March 2025, the state-of-the-art performance on SWE-bench Verified is achieved by the W&B Programmer O1 crosscheck5 model, which resolves 64.60% of the tasks. This model surpasses previous leaders, including Blackbox AI Agent at 62.80% and the CodeStory Midwit Agent with swe-search at 62.20%[1].

    SWE-bench Verified is a subset of the original SWE-bench dataset, consisting of 500 samples that have been human-annotated to ensure they are well-defined and solvable. This refinement aims to provide a more accurate evaluation of AI models' capabilities in resolving real-world software engineering tasks[2].

    Notably, Anthropic's Claude 3.5 Sonnet achieved a 49% resolution rate on SWE-bench Verified in January 2025, marking a significant improvement over its previous performance[3].

    These advancements reflect the rapid progress in AI-driven software engineering tools, with models continually improving their ability to autonomously address complex coding challenges.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.