Robocurve · 2026-09-04 · major
GPT-6 Astra on robot arms — 95% on a block task, Claude Fable 5.1 got 40%
Robocurve, an independent robotics benchmark group, gave GPT-6 Astra and Claude Fable 5.1 control of the same robot arms. Astra put the red block in the bowl 19 times out of 20; Fable 5.1 managed 8, at over twice the cost per run.
An independent benchmark put three frontier models in charge of the same robot arms and counted what they actually picked up.
Quick facts
| Publisher | Robocurve, an independent public benefit corporation |
|---|---|
| Models tested | GPT-6 Astra, Claude Fable 5.1, Claude Fable 5 |
| Hardware | Bimanual I2RT YAM arms, 6-DoF per arm, parallel-jaw grippers |
| Trials | 20 per model per task, two tasks |
| Settings | Medium thinking effort, 20-call budget, 25% speed cap |
| Harness | Inspect Robots, MIT licensed |
| Cost per bowl run | $0.94 Astra vs $2.12 Fable 5.1 (list price) |
Benchmarks
What is it?
Robocurve gave GPT-6 Astra, Claude Fable 5.1 and Claude Fable 5 control of one pair of robot arms and ran 20 trials each on two pick-and-place tasks. On the easy task — put a red block in a bowl — Astra finished 19 of 20 runs, against 8 for Fable 5.1 and 1 for Fable 5. On the harder task, fitting a round puzzle piece into a matching groove, Astra and Fable 5.1 both finished only 2 of 20.
How does it work?
Each turn the model sees three camera views — top, left wrist and right wrist — plus the arms' proprioceptive state, then replies with absolute end-effector poses: x, y, z, yaw, pitch, roll and a gripper value for each arm. The rig is a bimanual I2RT YAM setup with six degrees of freedom per arm and parallel-jaw grippers. Every model ran under the same Inspect Robots agent policy at medium thinking effort, with a 20-call budget and a 25% speed cap.
Why does it matter?
Robot-arm control is where a vision-language model stops being graded on text and starts being graded on whether the block ends up in the bowl. Speed and price moved with accuracy here: 2.5 minutes and $0.94 per bowl run for Astra against 6.8 minutes and $2.12 for Fable 5.1. The puzzle numbers are the other half of the story — both models miss nine times out of ten on a task that needs real precision, so general-purpose models are still a long way from dependable manipulation.
Who is it for?
robotics engineers and evaluation researchers
Frequently asked questions
- Did GPT-6 Astra beat Claude Fable 5.1 on both tasks?
- No. GPT-6 Astra won the easy task by a wide margin, finishing 19 of 20 block-in-bowl runs against Claude Fable 5.1's 8. On the harder puzzle task the two tied: both placed the piece in only 2 of 20 trials. Astra was faster and cheaper on both, but not more accurate on the precision task.
- How much did each robot-arm run cost?
- Robocurve reports list prices per run. On the bowl task GPT-6 Astra cost $0.94 against Claude Fable 5.1's $2.12 and Claude Fable 5's $2.69. On the puzzle task Astra cost $1.36, Fable 5.1 $2.18 and Fable 5 $2.63. Anthropic requests ran without prompt caching, while OpenAI cached about a fifth of Astra's input automatically, which Robocurve did not discount.
- What caveats does Robocurve list for these results?
- Robocurve states five. Astra's trials ran two days after the Fable trials rather than interleaved. The bowl comparison used a different rig because the original was unavailable. Grading was operator-judged with the model known, so unconscious bias is possible. Objects were reset by hand between trials, and every model ran at medium reasoning effort only.
- Can I run this benchmark on my own robot?
- Yes, if you have compatible hardware. The Inspect Robots harness Robocurve used is open source under MIT and lets you define a robotics benchmark once, then run any LLM or vision-language-action policy against a real arm, a humanoid, or a simulator, with grader scores, full transcripts and Rerun visualization. The project describes itself as alpha.
Try it
https://github.com/robocurve/inspect-robots