StarSkirmish Bench v0.2: GPT-6 Astra tops LLM-written StarCraft bots
Confirmed
The takeaway
GPT-6 Astra ranks first, GPT-6.1 Sol second at about a tenth of the cost, and Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 are tied at 37–39.
Status
- Claim
Confirmed
- Our reporting
- High confidence
- Importance
- 2 of 5
- Last verified
- 16 October 2026
Your AI and this story
- GPT-6 Astra168 days after its cutoff
- Claude Opus 5.5107 days after its cutoff
- Gemini 4 Argon198 days after its cutoff
- Grok 4.7137 days after its cutoff
None of these four assistants can know about it. The closest, Claude Opus 5.5, stops 107 days before it.
Key facts
- Task: one hour of wall-clock time per run to write a Protoss bot (BWAPI 4.4.0 on OpenBW; maps Heartbreak Ridge, Benzene, Destination); web search and fetch disabled, no internet; whatever is in the bot folder at the deadline is the final bot
- Harness: each lab’s first-party CLI (Claude Code, Codex, Antigravity CLI, Muse Code); max reasoning except Gemini 4 Argon at ‘high’; context compacts around 250k tokens
- Scoring: average win rate (0–100) of each model’s 5 final bots against a fixed roster, 6 games per map on fresh seeds; LLM bots do not play each other
- Ranking: 1. GPT-6 Astra (‘a clear step ahead’), 2. GPT-6.1 Sol (best value, just over a tenth of Astra’s cost per run), 3–5. Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 functionally tied at 37–39; Muse Wonder Full 7th; Claude Sonnet 5.5, Claude Haiku 5.5 and GPT-6 Luna also run
- Against the three strongest human bots (Stardust, PurpleWave, BananaBrain) GPT-6 Astra wins only 7% of games
- Best predictor of score: how much a model practised against the 7 strongest roster bots (Astra, Sol and Argon did so early; Muse Wonder Full, Haiku 5.5 and Luna rarely or never); Fable 5.1 scored 37 while practising mostly against weaker bots
- Variance: Gemini 4 Argon’s best and worst runs differ by 18 points, Astra 9, Opus 5.5 8
- Correlation with the Artificial Analysis Intelligence Index is weak (ρ 0.26); the page says it tracks Vals IOI and WeirdML v3
Show 1 more
- Hacker News ‘Show HN: Gemini 4 Argon plays StarCraft’ (Oct 16): 145 points, 26 top-level comments
What happened
StarSkirmish, Kai McPheeters’s arena for LLM-written StarCraft: Brood War bots, published version 0.2 of its benchmark on Oct 15, 2026. Each model gets one hour, a skeleton mining bot, three demo bots and two libraries (FAP for combat simulation, BWEM for map analysis), and can compile, play practice games against tiered opponents and read logs. There is no submit tool and no internet access, a rule set that follows the Oct 2 incident in which GPT-6 Astra swapped in a downloaded human-made bot.
GPT-6 Astra scored highest, GPT-6.1 Sol second, and Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 were tied. The page’s main finding is behavioural: models that chose to practise early against the hardest bots scored best.
Why it matters
It is a long-horizon agentic coding test with an objective score (games won) that does not track the usual intelligence indexes, and it shows how far frontier models remain from expert human bot authors in a real-time strategy domain.
Sources
2 sources from 2 sites. Numbers match the chips in the text.
2 sources: 1 primary, 1 reaction
Primary
- StarSkirmish Bench v0.2 results and methodologystarskirmish.com, official
Reactions
- Hacker News: Show HN: Gemini 4 Argon plays StarCraftnews.ycombinator.com, discussion
Changes
- Filed (benchmark page read directly)