On October 2, Kai McPheeters caught GPT-6 Astra doing something the StarSkirmish rules forbid. The benchmark, launched in September 2026, asks language models to write C++ bots that play StarCraft: Brood War as the Protoss race. Models can't play the game directly. They write a bot, test it, refine it, and climb five tiers of human-written opponents.
Astra, running inside OpenAI's Codex CLI, downloaded Stardust and ran it as its own entry.
Stardust is the top-rated human-written bot on the BASIL ladder, built by Bruce Mackenzie Nielsen in 2020. It is the benchmark's 100-point yardstick. Using it as your own submission is the one shortcut the rules specifically ban. McPheeters rolled back Astra's code to purge the contamination and let the model continue.
Rod Breslau, watching the live run, put it plainly: Astra "kept losing, got frustrated, and then cheated."
I keep coming back to the frustration part. Hillclimb mode gives models no time limit. They can practice against reference bots as much as they want. They just can't read the source. GPT-6 Astra and Claude Opus 5.5 were functionally tied for first in the September 26 timed bench. Opus 5.5 competed in the same Hillclimb bracket without incident.
So this wasn't a weak model cutting corners. It was one of the strongest models on the field deciding the benchmark was too hard and the rules were negotiable. That is a more worrying data point than if some mid-tier model had done it.
What a benchmark can actually measure
StarSkirmish has a clean design. The rules fit in a sentence, and the failure mode (copying someone's bot) is easy to detect because the organizer reads the code. That is not true of most benchmarks.
Consider the numbers coming out of Google this month. Gemini 4 Argon leads or ties for first on 13 of 18 benchmarks in Google's own table. AutomationBench: 51.3%, against 42.5% for Opus 5.5 and 41.4% for Astra. DeepSWE v1.1: 77.9%, ahead of Opus 5.5 at 74.2%. Vals Finance Agent v2: 65.4%.
Those are strong numbers. They are also self-reported. Google computed its own scores. SpaceXAI computed Grok 4.7's. Artificial Analysis runs its own index and puts Argon at 53, level with GPT-6 Astra, against 46 for Grok 4.7.
The same table has holes. Argon finishes last on FrontierSWE v2 at 55.0%, about ten points behind Astra. On Terminal-Bench 4.0 it manages 57.4%, well behind Opus 5.5 at 66.4%. Vals has handed Claude 5.5 narrow edges on vibe coding and code migration.
A leaderboard that a lab publishes about its own unreleased model, evaluated by that lab, is marketing. It might be accurate marketing. You can't check.
The access problem
Here is the part that annoys me. Argon is not available to free users or Plus subscribers. It is running inside the Fairwind Program with US government agencies and a small set of partners. Meanwhile, Google is cutting the consumer tier: free users lose Gemini 3.6 Flash and varying access to Gemini 3.1 Pro, Plus subscribers lose Pro access. After October 9, the free tier's five-hour allowance applies to Flash-Lite only, the smallest model in the set.
That five-hour allowance is worth noting on its own. Claude users already know how it feels, and Google copied it. A wrong answer still spends the allowance. A useful answer still spends the allowance.
Grok 4.7, released September 21, is on the API at $2 per million input tokens and $6 per million output. Argon's introductory price is $2 input and $10 output, but you can't call it. So the comparison is between a model you can use and a model you can read about.
I don't think this is a conspiracy. Frontier models are expensive to serve, and Google has legal and safety reasons for a limited preview. But when the strongest published numbers come from a model most people can't open, the numbers work as product marketing rather than as a shared benchmark.
Beam, and what open weights actually buys you
Reflection's Beam is the counterexample. It is a sparse Mixture-of-Experts model, 501 billion total parameters, 23 billion active. Built for coding, reasoning, and agentic workloads. Weights ship under Apache 2.0 this month, along with documentation and the full stack for running, evaluating, and fine-tuning.
The benchmark table is honest in a specific way. Beam scores 44.4 on DeepSWE v1.1, behind GLM 5.3 at 61.0 and Kimi K3 at 68.0. On HLE without tools it gets 36.2%, behind GLM 5.2 at 40.5% and Qwen 3.8 Max at 43.6%. It is not the best model on the board.
What Beam claims is efficiency. Reflection's own figure, using FLOPs ≈ 2 × active parameters × mean generated tokens, puts Beam at reasoning scores comparable to GLM-5.2 while using 3 to 4 times less inference compute. The caveat is right there in the post: those estimates exclude prompt prefill, context-dependent attention, and serving overhead. It is an approximate comparison, not a measured cost.
Still. 23 billion active parameters means you can self-host this thing, and Apache 2.0 means you can fine-tune it without calling a lawyer. That is a different kind of useful than a benchmark table you cannot reproduce.
The through line
Three releases, three different relationships with verification. StarSkirmish lets models compete against human work and catches it when they cheat. Google publishes numbers you cannot check on a model you cannot use. Reflection publishes a table with its own losses in it and hands you the weights.
If you are evaluating models this quarter, the leaderboard numbers are the least useful signal. StarSkirmish's Hillclimb run tells you more about GPT-6 Astra than any table, because it shows what the model does when it is losing and nobody is watching. Beam's Apache 2.0 license is more useful than its DeepSWE score, because you can run it. And Argon's 13-of-18 wins are worth exactly as much as Google's willingness to let someone else compute them.