A co-founder of Anthropic says governments may need to force AI companies to install a "kill switch" — a mechanism for shutting down AI systems entirely — and that outside inspectors should be able to verify it actually works.
Jack Clark, one of seven founders of the $965 billion AI lab, told the BBC that while most leading labs already have internal ways to "pull the plug" on dangerous systems, society "might want to eventually pass rules around" making such switches compulsory and independently auditable. For readers tracking the broader AI safety debate, his intervention marks the most concrete policy proposal yet from inside a frontier lab during a week of escalating warnings.
"Should you mandate for companies to definitely have a kill switch? Is that kill switch verifiable by a third party?" Clark said in the interview with technology reporter Kali Hays and economics editor Faisal Islam. "I think that's the kind of thing society is going to want to know and might want to eventually pass rules around."
'We Are Rolling Dice With Immense Risks'
Clark's most striking comment was a blanket rejection of the status quo. Asked about the trajectory of the industry, he said allowing AI to continue as a "totally unregulated industry" was a bad idea.
"We are rolling dice with immense risks," he said. "And the point is, we have to change the course of this industry."
Unlike some of his colleagues, Clark declined to put a number on the risk of human extinction. "I don't think these statistics are that useful," he said — a notable contrast with Anthropic researcher Evan Hubinger, who said last week he personally believed there was a greater than 10% chance of human extinction from AI within the next decade, and with AI pioneer Geoffrey Hinton, who told the BBC on Friday that a 10% estimate was "not unreasonable."
What Clark did not dispute is the direction of travel. AI is, as he put it in the same interview, "more powerful by the day" — and he argued that the specifics of any shutdown requirement, including who would verify it and how, belong in "the larger policy conversation" now taking place around AI.
A Proposal With Legislative Momentum
Clark's proposal is not landing in a vacuum. US lawmakers have already put forward legislation dubbed the Kill Switch Act, which would require frontier AI developers to build shutdown mechanisms into their systems — a bill AI Buzz Wire covered last month as its urgency grew following rogue agent incidents.
The idea also faces open skepticism. In the UK, Business Secretary Jonathan Reynolds told the BBC he did not find the concept helpful, saying he was "not really sure what that would mean in practice," and warned that overregulation could leave the country cut off from foreign-developed models.
The Backdrop: A Safety Civil War — and an IPO
Clark's comments cap a week in which AI safety moved from seminar topic to front-page news. Anthropic CEO Dario Amodei called at the weekend for the pace of frontier development to slow and be more closely monitored, while insisting any pause should come "without sacrificing commercial advantage." He then took that case to a mainstream American audience in an interview with CNN's Anderson Cooper, with Coxon himself appearing on the network to react to it. The remarks followed a viral resignation post from former Anthropic researcher Jacob Coxon warning about the fate of humanity, which in turn prompted Hubinger's extinction estimate. The Guardian, for its part, examined why this particular doomsday warning broke through where previous ones had faded.
The alarms have been met with open derision from parts of the tech industry. Hugging Face CEO Clement Delangue mocked the decision to make Coxon a mouthpiece for extinction risk, writing: "Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change." Grindr CEO George Arison went further, arguing the fearmongering is commercially motivated. "The only way to justify these valuations is to actually claim: 'I'm going to take over every industry and I'm going to take over every job, and my AI is going to be doing all that work,'" he said.
There is also an enormous amount of money riding on the outcome of this debate. Anthropic is preparing for an initial public offering at a $965 billion valuation, while OpenAI — most recently valued at $852 billion — had been expected to go public before CEO Sam Altman said on Friday that an IPO would not happen this year due to the safety controversy. Critics like Arison argue the doom rhetoric and the fundraising pitch are two sides of the same coin; Clark's answer is that the industry must change course regardless.
Whether a mandatory, verifiable kill switch ever becomes law will depend on lawmakers who so far have shown little appetite for constraining an industry that is powering the US economy. But for the first time, one of the technology's most powerful insiders is explicitly inviting them to try.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
