OpenAI · 2026-07-29 · major
OpenAI: two API settings tripled GPT-5.6 Sol on ARC-AGI-3 — 13.3% to 38.3%
OpenAI reran GPT-5.6 Sol on ARC-AGI-3's public task set with two Responses API settings — retained reasoning and compaction — turned on. Score jumped from 13.3% to 38.3% and used about one-sixth as many output tokens.
GPT-5.6 Sol jumped from 13.3% to 38.3% on ARC-AGI-3's public task set once OpenAI turned on the same Responses API settings ChatGPT already uses.
Quick facts
| Maker | OpenAI |
|---|---|
| Model | GPT-5.6 Sol |
| Benchmark | ARC-AGI-3 public task set |
| Score change | 13.3% → 38.3% |
| Output tokens | ~6x fewer |
| Settings enabled | Retained reasoning + compaction |
| Published | July 29, 2026 |
Benchmarks
| GPT-5.6 Sol (official harness) | 13.3% | |
|---|---|---|
| GPT-5.6 Sol (retained reasoning + compaction) | 38.3% |
What is it?
OpenAI's post argues that ARC-AGI-3 scores heavily depend on the surrounding harness, not the model alone. With the official evaluation, GPT-5.6 Sol scored 13.3% on the public task set. With two settings — retained reasoning and compaction — enabled through the OpenAI Responses API, the same model scored 38.3% and burned roughly one-sixth as many output tokens per game.
How does it work?
Retained reasoning preserves GPT-5.6 Sol's private chain-of-thought across steps of a task instead of erasing it after each move. Compaction summarizes older context when history overflows the window rather than truncating it. The official ARC-AGI-3 harness did neither, so the model kept losing its own scratchwork mid-puzzle and had to restart its reasoning constantly. Both settings are on by default inside ChatGPT and Codex.
Why does it matter?
ARC-AGI-3 is one of the highest-signal reasoning benchmarks in the AI industry, and a 3x swing from a harness tweak reframes what those numbers mean. OpenAI's implicit ask is that benchmark tables report the harness used, not just the model — otherwise a chart is silently ranking eval setups. The blog also gives developers a concrete recipe to squeeze better multi-step agent performance out of GPT-5.6 Sol.
Who is it for?
AI evaluators, benchmark maintainers, and developers building agentic workflows on GPT-5.6 Sol
Frequently asked questions
- What are retained reasoning and compaction in the OpenAI Responses API?
- Retained reasoning keeps GPT-5.6 Sol's private chain-of-thought across steps in a task instead of throwing it away after each response. Compaction summarizes older context when the running history overflows the window, rather than truncating and losing whatever fell out. Both are Responses API options ChatGPT and Codex already use by default.
- Why did ARC-AGI-3 score GPT-5.6 Sol so low at first?
- The official ARC-AGI-3 harness discarded GPT-5.6 Sol's private reasoning after every move and dropped older actions once history exceeded the window. GPT-5.6 Sol therefore kept restarting its train of thought mid-puzzle, so its 13.3% score reflected the harness's rules more than the model's real reasoning ability, OpenAI argues.
- Does this mean ARC-AGI-3 numbers can't be compared across models?
- OpenAI's takeaway is that ARC-AGI-3 measures the model plus the technical setup around it, not the model alone. Runs that differ in whether reasoning traces are kept or truncated can produce very different scores — so a fair comparison requires publishing which harness settings each entry used, not just the top-line percentage.
- How can developers enable the same two settings?
- The two flags live in OpenAI's Responses API and are the defaults inside ChatGPT and Codex. Third-party evaluators calling GPT-5.6 Sol directly need to opt into the same Responses API behavior — keeping reasoning between turns and enabling automatic context compaction — to see similar gains on multi-step agent tasks like ARC-AGI-3.
Try it
In the OpenAI Responses API, keep prior reasoning items across turns and enable context compaction — the same defaults ChatGPT uses