Cognition · 2026-06-08 · major
Cognition Launches FrontierCode — 150-Task Benchmark Grades Whether Coding Agents Produce Mergeable Pull Requests, Claude Opus 4.8 Tops the Hardest Diamond Tier at 13.4% While GPT-5.5 Hits 6.3% and Gemini 3.1 Pro 4.7%
Cognition unveiled FrontierCode, a 150-task benchmark that grades coding agents on whether their code would actually get merged into production repos. Built with 20+ open-source maintainers, each spending 40+ hours per task. Claude Opus 4.8 leads at 13.4% Diamond.
First coding benchmark grading whether AI agents write code maintainers would actually merge — not just code that passes tests.
Key specs
| Diamond tasks | 50 |
|---|---|
| Main tasks | 100 |
| Extended tasks | 150 |
| Claude opus 4.8 diamond | 13.4% |
| Gpt 5.5 diamond | 6.3% |
| Gemini 3.1 pro diamond | 4.7% |
| False positive cut vs swe bench pro | -81% |
What is it?
A 150-task software-engineering benchmark from Cognition, the team behind the Devin agent. Each task was hand-crafted by maintainers of 20+ flagship open-source repos who spent 40+ hours per task writing rubrics that match how they would actually review a pull request. The benchmark ships in three nested tiers: Extended (all 150 tasks), Main (the 100 hardest), and Diamond (the 50 hardest).
How does it work?
Every task ships with maintainer-written rubrics that cover behavioral correctness, test quality, scope discipline, mechanical cleanliness, and adherence to codebase style. Grading uses an ensemble of techniques: classical unit tests for behavior, reverse-classical tests that catch agents who delete failing assertions, code-scope validation that flags unrequested changes, and adaptive grading that re-runs tests after patches. Tasks are kept private to avoid contamination, with evaluation offered to model creators on request.
Why does it matter?
Public coding benchmarks like SWE-Bench keep saturating while real agent-authored PRs still get rejected for style violations, scope creep, and missing tests. FrontierCode reports an 81% lower false-positive rate than SWE-Bench Pro on its quality control, and the Diamond tier remains unsaturated: even Claude Opus 4.8 — the top model — scores 13.4%. That headroom gives agent labs a fresh, harder target to optimize against.
Who is it for?
Coding-agent labs, evaluation researchers, and engineering leaders comparing models for production use.