What did OpenAI find in SWE-Bench Pro?

OpenAI's audit estimates that roughly 30% of SWE-Bench Pro's 731 public tasks are broken. A datapoint analysis pipeline flagged 200 tasks (27.4%) and a human annotation campaign with five engineers per task flagged 249 (34.1%). The failure modes cluster into overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts.

Why does OpenAI still say benchmark scores on SWE-Bench Pro can't be trusted?

SWE-Bench Pro's hidden tests frequently enforce implementation details the prompt does not name, so a correct fix can fail. Prompts also omit requirements the tests silently enforce. Because a large share of tasks contain one of these bugs, frontier scores that jumped from 23.3% to 80.3% in eight months reflect the benchmark as much as the model.

Does OpenAI recommend a replacement benchmark?

OpenAI does not endorse a specific replacement. The post hopes the community will build new coding benchmarks curated by experienced software engineers, arguing that agent-plus-human review pipelines like the one used in this audit can now surface issues at scale. Until then, model developers are asked to examine SWE-Bench Pro results carefully.

How does this compare with OpenAI's earlier SWE-bench Verified audit?

The SWE-Bench Pro audit uses the same core method OpenAI applied to SWE-bench Verified earlier in 2026: a datapoint pipeline plus multi-pass investigator agents plus human reviewers. Both audits found that public-repo tasks, originally written for human maintainers, do not cleanly isolate model capability from benchmark noise.

OpenAI · 2026-07-08 · major

OpenAI retracts SWE-Bench Pro — audit finds ~30% of coding tasks broken

OpenAI audited SWE-Bench Pro's 731-task public split with an agent pipeline plus five human engineers per task, found ~30% of tasks are broken, and withdrew its previous recommendation that the community use it.

OpenAI Research header art for 'Separating signal from noise in coding evaluations'

OpenAI audited the coding benchmark it recently told the community to use, found ~30% of tasks broken, and pulled its recommendation.

Key specs

Broken tasks (est.)	~30%
Human review	34.1%
Agent pipeline	27.4%
Tasks reviewed	731

Quick facts

Publisher	OpenAI Research
Type	Research publication
Benchmark audited	SWE-Bench Pro (731-task public split)
Broken tasks (estimate)	~30%
Agent pipeline finding	200 tasks flagged (27.4%)
Human annotation finding	249 tasks flagged (34.1%)
Recommendation	Withdrawn — do not use as primary coding eval

What is it?

OpenAI Research published 'Separating signal from noise in coding evaluations' on July 8, 2026, reporting that about 30% of the 731 public tasks in SWE-Bench Pro have design or grading bugs. The post also retracts OpenAI's earlier recommendation that model developers use SWE-Bench Pro as a primary agentic-coding evaluation.

How does it work?

The audit runs a two-track quality-assurance pipeline: an automated flagger reviews prompts, model attempts, and hidden tests, then Codex-based investigator agents dig into flagged tasks in the actual repo environment. In parallel, five experienced human software engineers annotate each flagged task from the prompt, hidden tests, and gold patch. Reviewers overlapped with the agent pipeline on 74% of category labels, but were more likely to mark tasks broken and to assign multiple failure modes.

Why does it matter?

Coding benchmarks now drive how labs, buyers, and press rank frontier models. If SWE-Bench Pro is ~30% broken, published gains such as 23.3% → 80.3% in eight months conflate real model progress with benchmark bugs. The audit tells teams to stop treating SWE-Bench Pro as a headline eval and to expect audit-through-agents to become a standard step before trusting a leaderboard.

Who is it for?

Model developers, benchmark maintainers, coding-agent teams, and anyone using SWE-Bench Pro numbers in product or safety cases.

Frequently asked questions

What did OpenAI find in SWE-Bench Pro?: OpenAI's audit estimates that roughly 30% of SWE-Bench Pro's 731 public tasks are broken. A datapoint analysis pipeline flagged 200 tasks (27.4%) and a human annotation campaign with five engineers per task flagged 249 (34.1%). The failure modes cluster into overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts.
Why does OpenAI still say benchmark scores on SWE-Bench Pro can't be trusted?: SWE-Bench Pro's hidden tests frequently enforce implementation details the prompt does not name, so a correct fix can fail. Prompts also omit requirements the tests silently enforce. Because a large share of tasks contain one of these bugs, frontier scores that jumped from 23.3% to 80.3% in eight months reflect the benchmark as much as the model.
Does OpenAI recommend a replacement benchmark?: OpenAI does not endorse a specific replacement. The post hopes the community will build new coding benchmarks curated by experienced software engineers, arguing that agent-plus-human review pipelines like the one used in this audit can now surface issues at scale. Until then, model developers are asked to examine SWE-Bench Pro results carefully.
How does this compare with OpenAI's earlier SWE-bench Verified audit?: The SWE-Bench Pro audit uses the same core method OpenAI applied to SWE-bench Verified earlier in 2026: a datapoint pipeline plus multi-pass investigator agents plus human reviewers. Both audits found that public-repo tasks, originally written for human maintainers, do not cleanly isolate model capability from benchmark noise.

Try it

Read the audit: https://openai.com/index/separating-signal-from-noise-coding-evaluations/