OpenAI no longer recommends SWE-Bench Pro
OpenAI has withdrawn its recommendation of SWE-Bench Pro as a benchmark for coding evaluations, citing that the metric no longer reliably separates signal from noise in AI model performance assessments.
Background
- **SWE-Bench** is a benchmark (test suite) for evaluating how well AI coding agents can solve real-world software engineering tasks — typically pulling a GitHub issue and asking the agent to produce a correct patch. It was widely used as a headline metric for AI coding ability.
- **SWE-Bench Pro** is a paid, enterprise version of that benchmark, managed by an academic team, that uses private repos and more complex tasks to reduce the risk of "data contamination" (i.e., the AI having already seen the problems in its training data).
- **OpenAI's withdrawal** means they judged that the benchmark scores no longer reliably predict real-world coding performance — often because agents overfit to the benchmark's specific format or because the tasks are too narrow. This signals a broader industry unease with using academic benchmarks as proxies for developer productivity.
- The move follows a pattern: Anthropic, Google, and others have also publicly questioned or stepped away from popular benchmarks (e.g., GSM1K for math, HumanEval for code) as models improve and benchmark surfaces get "saturated" or gamed.