Decentralized AI lab Prime Intellect has published the results of a large-scale experiment testing how well today's frontier AI models can perform autonomous machine learning research — and the best of them came remarkably close to matching the human world record on a celebrated community benchmark.
The project, dubbed NanoGPT Speedrun Frontier and published on August 22, consisted of 153 autonomous runs across 18 frontier models attempting the nanoGPT optimizer speedrun — a competition in which researchers search for the most efficient way to train a small language model to a fixed quality target. The story quickly climbed the front page of Hacker News, where it drew dozens of comments debating what the results say about AI's capacity for genuine scientific discovery. For more context on this story, see our ongoing artificial intelligence updates.
What the NanoGPT Speedrun Actually Is
The benchmark comes from the modded-nanogpt repository maintained by Keller Jordan and a long list of community contributors. The challenge is deceptively simple to state: using 8 NVIDIA H100 GPUs, train a language model that reaches 3.28 cross-entropy loss on the FineWeb validation set — the performance level Andrej Karpathy's original GPT-2 replication reached after 45 minutes — as fast as possible.
Through hundreds of community contributions over the past two years, the human record has been driven down to under 75 seconds and under 400 million training tokens, a more than 30-fold improvement over the original recipe. The winning solutions read like a catalog of modern ML engineering: the Muon optimizer, rotary embeddings with QK-Norm, ReLU-squared activations, FP8 quantization on attention and MLP passes, Flash Attention 3 with sliding-window patterns, and dozens of other techniques stacked on top of each other.
Prime Intellect's test used the benchmark's optimizer track, which — per the repository's documentation — minimizes the number of training steps needed to hit the target loss, subject to a fixed architecture, dataset, and batch size, with an effectively unlimited wall-clock budget. That makes it a pure test of algorithm discovery rather than raw hardware speed.
How the Experiment Worked
Each of the 18 models was given the task autonomously: propose changes to the training recipe, run experiments, inspect the results, and iterate — with a fixed verification script and fixed random seeds ensuring that a claimed improvement is real rather than a lucky run. As one Hacker News commenter summarized it, the models were asked to research how to improve a small model's training, repeatedly trying changes, testing them, and using the results to decide what to try next.
The models worked through a variety of agent harnesses — including claude-code, OpenAI's codex, kimi-code, grok-cli, qwen-code, and Prime Intellect's own prime-agent — and the team tracked every run's token consumption, experiment count, tool calls, and elapsed agent time. In total, the experiment consumed hundreds of millions of tokens per top model and billions across the full grid.
The Leaderboard: Fable 5 Leads the Pack
The headline result: the model identified as Fable 5, running through the claude-code harness, closed 81.7 percent of the gap between the baseline recipe and the human record, with a best validated result of 2,726 on the leaderboard's metric. Getting there took 8.7 days of cumulative agent time, roughly 800 million tokens, 811 experiments, and around 3,000 tool calls.
The chasing pack was led by Opus 5, which closed 53.6 percent of the gap, followed closely by Kimi K3 at 52.2 percent when driven by Prime Intellect's prime-agent harness (and 45.8 percent via kimi-code). Opus 4.8 managed 39.4 percent, several GPT-5.6 variants landed between roughly 11 and 36 percent, Sonnet 5 closed 26.8 percent, and Grok 4.5 and Qwen3.8 Max each reached about 24.6 percent.
Further down, DeepSeek V4 Pro closed 12.3 percent of the gap, GPT-5.5 reached 8.1 percent, and the older Kimi K2.7 finished at 7.2 percent. Notably, one recently released model — GLM 5.3 — was still running at publication time with no validated record yet, a reminder that the benchmark punishes models that cannot reliably validate their own improvements.
Why It Matters
Benchmarks like this matter because they measure something coding leaderboards cannot: the ability to run a genuine research loop — hypothesis, experiment, analysis, revision — over days rather than minutes. That capability is the foundation of the emerging field of autonomous AI research, in which agents are tasked not with writing code to spec but with discovering things nobody has shown them.
The spread in results is itself informative. Almost every model in the experiment found the same core winning ideas, according to the project's writeup — what separated the best runs was what an experiment leaves behind: stronger agents preserved weak signals in noisy results long enough to validate them, and understood their own experimental data better.
Caveats From the Community
Hacker News commenters raised fair methodological points. Effort settings and harnesses varied across models — Fable 5 ran on a high reasoning setting while Opus 5 ran on max — and the arithmetic of 153 runs across 18 models implies some models received more attempts than others. Whether a model's ranking reflects its raw research ability or the quality of its harness remains an open question.
Still, the direction of travel is clear. When the fastest human hands took years of collective effort to reach the current record, the best autonomous agents are now closing most of that gap in a little over a week of compute time. The next question — whether any model can close the remaining 18.3 percent — may be answered sooner than expected.
For more on the benchmarks and research shaping the frontier of AI, follow AI Buzz Wire's continuing coverage.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
