SWE-bench Verified

A human-validated subset of 500 SWE-bench instances for reliable evaluation of coding agents and language models.
SWE-bench Verified is a human-filtered subset of 500 instances from SWE-bench, created in collaboration with OpenAI. Human annotators reviewed each instance to ensure the problem descriptions are clear, the test patches are correct, and the tasks are solvable given the available information. Read more in the OpenAI blog post.
The Verified leaderboard features results from a wide variety of AI coding systems, from simple LM agent loops to RAG systems to multi-rollout and review type systems.
While the full leaderboard compares arbitrary systems, we are also interested in evaluating language models directly. To make an apples-to-apples comparison of LMs easier, we evaluate all LMs using mini-SWE-agent in a minimal bash environment. No tools, no special scaffold structure; just a simple ReAct agent loop. These results represent the state-of-the-art LM performance when given just a bash shell and a problem.
On the leaderboard, use the Agent dropdown to select between the mini-SWE-agent results and the full leaderboard with all agents.
If you use SWE-bench Verified in your research, please cite our paper: