A benchmark that asks large language models to write competitive StarCraft bots in one hour has delivered its most detailed results yet, and the verdict is split: OpenAI's GPT-6 Astra dominates the AI field, but the strongest human-written bots still beat the best model 93% of the time.
StarSkirmish Bench v0.2, published October 15 by developer Kai McPheeters, evaluates how strong a StarCraft: Brood War bot each LLM can produce given a single hour of wall-clock time. Each model writes a Protoss bot in C++, which is then played against bots written by rival models and against established human-written competitors. The test is designed to probe long-horizon reasoning and agentic coding, since models must incorporate lessons from practice games to improve their creations. For more context on this story, see our ongoing AI news.
How the Benchmark Works
The rules are deliberately simple. One hour, one language, one race. Each model's score is the average win rate across five runs against a roster of competitive human-written bots and demo bots. Version 0.2 added a twist: every flagship model runs inside its own lab's first-party CLI coding harness, testing the full agentic stack rather than a stripped-down API loop.
There is one asymmetry, which the benchmark author notes upfront: Gemini 4 Argon runs at "high" — the highest reasoning effort available to it — while every other model runs at maximum.
GPT-6 Astra Leads, GPT-6.1 Sol Wins on Value
The results put GPT-6 Astra at the top of the leaderboard, which the writeup describes as a clear step ahead of GPT-6.1 Sol in second place. The margin matters for benchmark watchers: in most agentic coding suites, flagship models cluster within a few points of each other, so a leaderboard that separates the field into distinct tiers is doing more diagnostic work than a crowded top ten. Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 are functionally tied behind the two OpenAI models, all scoring between 37 and 39. Meta's Muse Wonder Full took seventh.
The cost analysis may be the more striking finding. Plotted against the average API cost of a one-hour run, GPT-6.1 Sol lands alone in the most attractive quadrant: second place at just over a tenth of GPT-6 Astra's cost per run. For teams weighing agent performance against budget, the gap between the two OpenAI models is the practical headline.
Human Bots Remain Far Ahead
Against the strongest human-written bots — Stardust, PurpleWave and BananaBrain — GPT-6 Astra wins only 7% of its games. The benchmark's author notes that the one-hour reasoning limit is a factor, and that tournament conditions for real human-written bots are significantly harder than the three two-player maps used in evaluation.
Still, the gap is a useful corrective to narratives of imminent AI dominance in software: on a task with clear win conditions, deep strategy and long planning horizons, decades of community engineering still stand well above what an hour of frontier-model coding can produce.
StarCraft has long served as a proving ground for game-playing AI precisely because it resists brute force. Real-time strategy demands simultaneous management of economy, scouting and combat under partial information, and the Brood War community has maintained competitive bot tournaments for years. That accumulated ecosystem is what makes the benchmark's human baseline meaningful — Stardust, PurpleWave and BananaBrain are not casual opponents but refined, tournament-hardened programs.
New Flagships Enter the Fray
Version 0.2 also served as an early scoreboard for freshly released models. Google's Gemini 4 Argon replaced v0.1's Gemini 3.8 Flash and placed third on debut, while Meta's Muse Wonder Full stepped in for Muse Spark 1.3 and finished seventh. GPT-6 Astra and Claude Opus 5.5 carried over from the earlier version's coverage.
The benchmark records an agent trace for each run, logging how each model spent its hour — a dataset that turns the leaderboard into a behavioral study of how different labs' models budget time between planning, coding and playtesting. Whether a model front-loads architecture decisions or iterates rapidly through buggy drafts is exactly the kind of process detail that aggregate scores hide, and it is where the most interesting cross-lab differences tend to surface.
Why It Matters
Agentic coding benchmarks are saturating fast, with flagship models clustering near the top of many software-engineering suites. StarSkirmish offers an alternative: an environment where the ceiling is set by human experts, progress is measured in win rates rather than percentage points of ticket resolution, and the hour-long time budget mirrors real-world constraints on agentic work.
The results struck a chord with developers. The Show HN post drew more than 200 upvotes, with commenters digging into the agent traces and the cost-efficiency plot. Benchmark discussions on Hacker News tend to split between methodology quibbles and capability hot takes; this one drew both, with the reasoning-effort asymmetry and the five-run sample size among the most scrutinized design choices. As reasoning effort levels rise and harnesses improve, the benchmark's author plans continued iterations — and for now, the human StarCraft bot community keeps its comfortable lead.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
