StarSkirmish · 2026-10-15 · notable
StarSkirmish Bench v0.2 — LLMs get one hour to code a StarCraft bot
StarSkirmish Bench v0.2 gives nine models one hour to write a StarCraft: Brood War bot in C++. GPT-6 Astra leads, GPT-6.1 Sol is second at about a tenth of the cost, and Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 tie at 37–39.
A long-horizon coding benchmark: write a StarCraft bot in an hour, then beat human-written bots with it.
What is it?
StarSkirmish Bench v0.2, published on 15 October 2026 by Kai McPheeters, tests how strong a StarCraft: Brood War bot an LLM can write in one hour of wall-clock time. Each model writes a Protoss bot in C++, starting from a mining-only skeleton plus demo bot source code and the FAP and BWEM libraries.
How does it work?
Models run in their lab's own CLI (Claude Code, Codex, Antigravity CLI, Muse Code) at maximum reasoning, with web access off, and can compile, play practice games and read logs. Each model's five final bots then play competitive human bots and demo bots on three ladder maps through OpenBW and BWAPI 4.4.0, and the score is the average win rate from 0 to 100.
Why does it matter?
The benchmark rewards learning from practice over a long task, not one-shot answers. Even the leader, GPT-6 Astra, wins only 7% of its games against the top human bots Stardust, PurpleWave and BananaBrain. Gemini 4 Argon was the least consistent model, with an 18-point gap between its best and worst runs.
Who is it for?
people comparing models on agentic coding