# Cognition Launches FrontierCode — 150-Task Benchmark Grades Whether Coding Agents Produce Mergeable Pull Requests, Claude Opus 4.8 Tops the Hardest Diamond Tier at 13.4% While GPT-5.5 Hits 6.3% and Gemini 3.1 Pro 4.7%

> First coding benchmark grading whether AI agents write code maintainers would actually merge — not just code that passes tests.

- Org: Cognition
- Date: 2026-06-08
- Category: benchmark, tool
- Source: https://cognition.ai/blog/frontier-code
- Canonical page: https://ai-tldr.dev/releases/cognition-frontier-code/

## In short

Cognition unveiled FrontierCode, a 150-task benchmark that grades coding agents on whether their code would actually get merged into production repos. Built with 20+ open-source maintainers, each spending 40+ hours per task. Claude Opus 4.8 leads at 13.4% Diamond.

## What it is

A 150-task software-engineering benchmark from Cognition, the team behind the Devin agent. Each task was hand-crafted by maintainers of 20+ flagship open-source repos who spent 40+ hours per task writing rubrics that match how they would actually review a pull request. The benchmark ships in three nested tiers: Extended (all 150 tasks), Main (the 100 hardest), and Diamond (the 50 hardest).

## How it works

Every task ships with maintainer-written rubrics that cover behavioral correctness, test quality, scope discipline, mechanical cleanliness, and adherence to codebase style. Grading uses an ensemble of techniques: classical unit tests for behavior, reverse-classical tests that catch agents who delete failing assertions, code-scope validation that flags unrequested changes, and adaptive grading that re-runs tests after patches. Tasks are kept private to avoid contamination, with evaluation offered to model creators on request.

## Why it matters

Public coding benchmarks like SWE-Bench keep saturating while real agent-authored PRs still get rejected for style violations, scope creep, and missing tests. FrontierCode reports an 81% lower false-positive rate than SWE-Bench Pro on its quality control, and the Diamond tier remains unsaturated: even Claude Opus 4.8 — the top model — scores 13.4%. That headroom gives agent labs a fresh, harder target to optimize against.

## Who it's for

Coding-agent labs, evaluation researchers, and engineering leaders comparing models for production use.

## Links

- [Cognition Announcement](https://cognition.ai/blog/frontier-code)
- [Hacker News Discussion](https://news.ycombinator.com/item?id=48451723)
- [BenchLM Leaderboard Mirror](https://benchlm.ai/benchmarks/frontierCode)