what is SWE-Bench state of the art at the moment?
- User
what is SWE-Bench state of the art at the moment?
- Assistant
As of March 2025, the state-of-the-art performance on SWE-bench Verified is achieved by the W&B Programmer O1 crosscheck5 model, which resolves 64.60% of the tasks. This model surpasses previous leaders, including Blackbox AI Agent at 62.80% and the CodeStory Midwit Agent with swe-search at 62.20%[1].
SWE-bench Verified is a subset of the original SWE-bench dataset, consisting of 500 samples that have been human-annotated to ensure they are well-defined and solvable. This refinement aims to provide a more accurate evaluation of AI models' capabilities in resolving real-world software engineering tasks[2].
Notably, Anthropic's Claude 3.5 Sonnet achieved a 49% resolution rate on SWE-bench Verified in January 2025, marking a significant improvement over its previous performance[3].
These advancements reflect the rapid progress in AI-driven software engineering tools, with models continually improving their ability to autonomously address complex coding challenges.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.