More than 20 researchers — among them Nobel laureate Geoffrey Hinton, fellow AI pioneer Yoshua Bengio, Anthropic co-founder Jack Clark and OpenAI chief scientist Jakub Pachocki — have warned governments to prepare for an AI "intelligence explosion" that they say could be the most consequential technological development in history, The Guardian reported Monday.
The paper, titled "What if automating AI R&D triggers an intelligence explosion", urges politicians to act before there is runaway progress in the technology. Its authors describe the scenario as a "dramatic AI-driven acceleration of AI progress, compressing advances that would otherwise take years into months or less". For readers tracking the latest AI research news, the report is notable less for its alarm than for who signed it: figures from across academia and rival frontier labs, jointly making the case that automated research is the scenario policymakers are least prepared for.
What an intelligence explosion would mean
The authors' central concern is the prospect of AI systems improving themselves without human intervention — a process known as recursive self-improvement, referred to broadly in the paper as automated AI research and development.
Once AI systems reach expert-level capabilities at AI R&D, the paper argues, a single developer could run a workforce equivalent to "millions" of top human researchers. Automating AI R&D is, in the authors' view, the most likely trigger of an intelligence explosion, because AI systems are already contributing to improving their own technology — and the improved systems they help create can be rapidly deployed once built.
"An intelligence explosion could be the most consequential technological development in human history, compressing years of progress into months or less, threatening human control over AI systems, and severely eroding checks on power within and between states, companies, and branches of government," the paper says, according to The Guardian.
Why the authors say the window is closing
The report's urgency rests on a timeline claim: AI systems that appear glitchy today could fully automate R&D projects that currently take human teams months by 2028.
"Productivity gains from AI R&D automation have not yet reached the threshold needed to trigger an intelligence explosion, but gains from newer systems are likely approaching that threshold," the authors write. "Once an intelligence explosion begins, the window for action may close."
The paper adds that "preliminary evidence suggests that a software-driven intelligence explosion is possible", which could lead to the "extremely rapid development of highly capable or superhuman AI systems".
The risks they flag
While an intelligence explosion could produce medical breakthroughs and rapid technological leaps, the authors outline three principal dangers:
- Runaway threats. Powerful systems could enable biological and cyber threats that develop faster than the measures needed to counter them.
- Lost control. As humans become less involved in AI R&D, they could lose the opportunity to direct and constrain the systems they build.
- Strategic instability. States could use an intelligence explosion to convert a modest lead — in cyberspace, for example — into a decisive national advantage.
The report is candid that these impacts are "uncertain". A technological breakthrough could take time to implement given supply chains for specialised materials or regulatory requirements, and AI systems could also accelerate the speed at which risks are mitigated.
What governments should do
The authors say preparing for an intelligence explosion "should be an urgent priority, including at the highest levels of government leadership", and lay out a set of candidate policies:
- Requiring transparent progress reports on AI-related R&D, including embedding independent auditors inside AI companies
- Limiting how fast an AI system can improve within a given time period
- Working with data centres to enable pausing certain AI R&D projects
- Ensuring automated AI R&D systems are fully isolated and cannot escape human control
- Creating "emergency response plans" for the range of scenarios an intelligence explosion could produce
The proposals extend ideas already moving through Western AI policy — mandatory incident reporting, third-party evaluation and compute governance — but apply them to a specific endpoint: the automation of research itself.
Why these authors, and why now
The signatories give the paper unusual weight. Hinton and Bengio are two of the "godfathers" of modern AI, recognised for foundational work on neural networks; both have become prominent voices warning about the technology they helped create. Clark co-founded Anthropic, and Pachocki is chief scientist at OpenAI.
Their warning lands amid measurable industry movement in that direction. Anthropic has said AI now produces 80% of its own code, while OpenAI uses autonomous AI agents in areas such as the training of new models, according to The Guardian. Both labs have also agreed to have independent evaluators assess their models after warning that AI development was reaching a critical pitch in terms of safety.
Whether governments will move at the pace the authors want is another question. But the paper's core argument — that the moment to build the brakes is before the car needs them — is now endorsed by the people closest to the machine.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
