How does SWE-Bench ProMax differ from SWE-bench Verified?

SWE-bench Verified asks agents to resolve issues, mostly in Python and mostly in small patches. SWE-Bench ProMax instead targets behavior-preserving refactoring across seven languages, at roughly 11.4 files and 261.6 lines per task. Its issue descriptions were rewritten from scratch and its tests manually audited, addressing quality problems documented in the older suite.

Which programming languages does SWE-Bench ProMax cover?

SWE-Bench ProMax spans seven languages with a fairly even split across its 170 instances: Python (29), TypeScript (28), Java (26), Go (23), C++ (22), Rust (22), and C (20). That balance is deliberate — most prior coding-agent benchmarks lean heavily on Python, which makes cross-language generalization hard to measure.

Where can you download SWE-Bench ProMax?

SWE-Bench ProMax is published as a Hugging Face dataset at swe-bench-promax/SWE-Bench-ProMax, totalling 13.1 MB. The main file is swe-bench-promax.json, with evaluation metadata in eval.json. Each instance carries the task description, repository metadata, a reference patch, and the evaluation test patch needed to score an attempt.

Why do models score so low on SWE-Bench ProMax?

The paper reports a best resolve rate of 41.2% across frontier models under two agent frameworks. Refactoring is harder than issue fixing because a correct answer must change many files at once and preserve existing behavior everywhere. Tasks with limited cross-file scope were filtered out, so nothing easy remains in the set.

SWE-Bench ProMax · 2026-08-10 · major

SWE-Bench ProMax — coding-agent benchmark where the best model scores 41.2%

SWE-Bench ProMax is a code-refactoring benchmark of 170 expert-curated tasks across seven languages. Each task changes 11.4 files and 261.6 lines on average, and the best frontier model resolves only 41.2% of them.

Hugging Face paper card for SWE-Bench ProMax, a multilingual code refactoring benchmark

A refactoring benchmark built to replace saturating SWE-bench tests, and frontier agents solve well under half of it.

Key specs

Best resolve rate	41.2%

Quick facts

What it tests	Large-scale, behavior-preserving code refactoring
Instances	170
Languages	Python, Java, TypeScript, Go, C, C++, Rust
Task size	11.4 files and 261.6 lines changed on average
Best result	41.2% resolve rate
Availability	Hugging Face dataset, 13.1 MB
Venue	COLM 2026

What is it?

Code refactoring, not bug fixing, is what SWE-Bench ProMax measures: 170 tasks drawn from real commits across Python, Java, TypeScript, Go, C, C++, and Rust. Every task demands coordinated edits that preserve behavior across many files, averaging 11.4 modified files and 261.6 lines — well beyond the single-file scale of earlier suites. The work was accepted to COLM 2026.

How does it work?

Curation is what separates SWE-Bench ProMax from its predecessors. The authors rewrote every issue description from scratch so the specification is precise and unambiguous, then manually reviewed each test suite to strip out tests that were too narrow (rejecting correct solutions) or too broad (checking requirements nobody stated). Tasks with too little complexity or too little cross-file reach were filtered out entirely.

Why does it matter?

Existing agent benchmarks are running out of headroom, and the paper cites an audit finding that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests, with frontier models able to reproduce gold patches verbatim from training data. SWE-Bench ProMax gives teams a harder, cleaner yardstick: the best model tested clears only 41.2%, leaving obvious room to measure real progress.

Who is it for?

teams building or evaluating coding agents

Frequently asked questions

How does SWE-Bench ProMax differ from SWE-bench Verified?: SWE-bench Verified asks agents to resolve issues, mostly in Python and mostly in small patches. SWE-Bench ProMax instead targets behavior-preserving refactoring across seven languages, at roughly 11.4 files and 261.6 lines per task. Its issue descriptions were rewritten from scratch and its tests manually audited, addressing quality problems documented in the older suite.
Which programming languages does SWE-Bench ProMax cover?: SWE-Bench ProMax spans seven languages with a fairly even split across its 170 instances: Python (29), TypeScript (28), Java (26), Go (23), C++ (22), Rust (22), and C (20). That balance is deliberate — most prior coding-agent benchmarks lean heavily on Python, which makes cross-language generalization hard to measure.
Where can you download SWE-Bench ProMax?: SWE-Bench ProMax is published as a Hugging Face dataset at swe-bench-promax/SWE-Bench-ProMax, totalling 13.1 MB. The main file is swe-bench-promax.json, with evaluation metadata in eval.json. Each instance carries the task description, repository metadata, a reference patch, and the evaluation test patch needed to score an attempt.
Why do models score so low on SWE-Bench ProMax?: The paper reports a best resolve rate of 41.2% across frontier models under two agent frameworks. Refactoring is harder than issue fixing because a correct answer must change many files at once and preserve existing behavior everywhere. Tasks with limited cross-file scope were filtered out, so nothing easy remains in the set.

Try it

https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax