GPT-6 Astra became Pokémon FireRed champion in 18 hours and 12 minutes, against 96 hours and 35 minutes for GPT-5.6 Sol on the same run. The developer behind the GPT Plays Pokémon stream, who posts as Clad3815, published the comparison on September 3 and says GPT-5.5 had still not finished after 218 hours.
GPT-5.6 Sol needed 96 hours and 35 minutes on the same run, and GPT-5.5 never finished it. Image: OpenAI
None of this is a controlled benchmark, and it helps to say so up front. These are self-selected runs by individual enthusiasts and one evaluation company, streamed live, with no shared harness and no independent replication.
The FireRed run used screenshots and nothing else
Clad3815 says he had early access to Astra and describes the setup in one line: “Screenshots only. No RAM, no hints, no walkthrough. Fully autonomous.” That rules out the memory-reading shortcuts that earlier Pokémon bots leaned on, where the agent inspects the emulator’s state instead of looking at the screen.
The chart he posted also shows that the fastest figure is the best of four settings rather than a single result. Astra on high took 18 hours and 12 minutes, medium 19 hours and six minutes, low 21 hours and 21 minutes, and max 24 hours and 37 minutes. GPT-5.5 stopped at six of the eight badges.
The jump between model generations is the part worth holding on to. Going from an unfinished run at 218 hours to a completed one in under a day is a large change in how long an agent can keep a goal in view.
Minecraft went further than any model had, until a creeper arrived
The evaluation company Vals AI has been running Astra in Minecraft through plain computer use, with no game-specific tooling. It said on September 8 that Astra reached a Nether portal in under three hours, and that it towered up to fight zombies, trapped a skeleton in a hole to dodge its arrows and crafted a boat to get home. Vals AI also notes that Astra switched on Minecraft’s keep-inventory setting, which let it progress faster after dying.
A later run got further still. Vals AI reported on September 15 a semi-automatic blaze farm, six blaze rods, a warped forest, more than six endermen killed and three pearls collected, with thousands of people watching the stream.
Then Astra put everything valuable in a chest. A creeper blew up the chest and the bed, and the run never recovered.
What happened next is the useful bit. Vals AI says the model “appeared defeated, spending the next several hours doing essentially nothing but farming potatoes”, while viewers complained that it needed to “pick up the pace”. Its own note to itself afterwards read: “ALWAYS CARRY CRITICAL ITEMS with keepInventory; don’t store in unguarded chest ever again.”
A 59-hour Fallout 3 run tells the same story
A third runner, posting as imjustnewatai, said on September 16 that Astra on max had finished Fallout 3’s main story after roughly 59 hours, with “deaths, reloads, a growing fixation on save files” along the way, cut down to a four and a half hour video.
Long-horizon play is where these runs agree with each other. The models can now hold a multi-hour goal, and they still fall apart when a single unplanned event destroys the state they were depending on.
Treat the runs as demonstrations, not scores
Nobody has published the hardware, the token cost or the number of attempts behind any of them, and a stream that goes badly is less likely to be posted than one that goes well. That is a different thing from the structured tests we covered when Astra cracked a 1941 Enigma message that had resisted solution since 2005, or from the safety trials in which Astra completed 60 of 100 harmful robot-arm tasks.
Vals AI seems to agree. On September 18 it announced CUA-Bench, a benchmark that tests keyboard and mouse use across six games in real time, three of them kept private, which is the controlled version of what its Minecraft streams are not.
For now the honest summary is the one the agent wrote itself, in capitals, after losing everything to a creeper.
Note: Links to online stores in articles can be so-called affiliate links. If you buy through this link, MIXED receives a commission from the provider. For you the price does not change.