Rep. Ro Khanna, a Silicon Valley Democrat, will introduce legislation that would ban AI models capable of recursively improving themselves until the federal government builds a regulatory system capable of overseeing them — the most sweeping AI safety proposal yet to land in Congress.
The bill, dubbed the Human Control Over AI Act, was shared exclusively with CNBC in summary form ahead of its introduction. It arrives amid escalating safety concerns that have already prompted the CEOs of OpenAI and Anthropic to call for slowing frontier model development. For more context on this story, see our ongoing latest AI developments.
"There's actually a civilizational extinction risk," Khanna said Monday in an interview with CNBC. "There's a safety risk of loss of control, and then there's a misuse risk, and we need to take both seriously."
What the Bill Would Ban
At its core, the Human Control Over AI Act would prohibit AI models that recursively self-improve or autonomously modify their own core objectives, containment mechanisms or shutdown controls — unless federal guardrails exist and a new government agency approves the activities.
In practical terms, that targets the capability that safety researchers fear most: systems that can redesign themselves in a self-directed cycle, leaving humans with diminishing oversight at each step. The ban would remain in place until regulators are equipped to evaluate the risks.
A New Federal Agency for Frontier AI
The bill would create a dedicated federal agency focused on the most powerful AI models — those developed by OpenAI, Anthropic, Google DeepMind and xAI — as those companies find themselves under intensifying scrutiny.
According to the summary shared with CNBC, the agency would be responsible for:
- Writing safety regulations for frontier AI systems
- Operating a licensing system to approve model training and deployment
- Conducting audits of frontier AI models
- Setting security standards for continuous testing and effective human control
Independent auditors would be embedded at every frontier laboratory and would report directly to the agency — a structural check designed to ensure that labs cannot simply mark their own homework. The agency would also set standards covering sandbox testing environments, air gaps that isolate testing systems from the internet, kill switches, and controls that prevent models from escaping lab settings and roaming the open internet.
Notably, the legislation would extend federal oversight to the advanced chips that frontier labs use, including mechanisms to monitor or control how those processors are deployed.
Criminal Penalties and Mandatory Insurance
The bill goes further than any previous proposal on penalties. It would criminalize what it defines as "crimes against humanity" for the deployment of AI models that result in the destruction of civilian populations.
Employees of AI companies would face criminal penalties for disabling safeguards, kill switches, logging or containment systems — or for knowingly deploying an unauthorized system. AI companies would also be required to carry extensive liability insurance before releasing their models, shifting the financial risk of failure onto the industry rather than the public.
On the geopolitical front, the bill calls on the administration to pursue enforceable agreements and targeted export controls designed to deter China and other adversaries from developing dangerous AI systems.
Built With the Safety Community, Not the Labs
Khanna was explicit about whose input shaped the bill. Rather than negotiating with the executives of the companies it would regulate, he said the legislation was modeled on conversations with AI safety organizations, including the independent nonprofits Model Evaluation and Threat Research, the Machine Intelligence Research Institute and Palisade Research.
"I want this to be a model to give voice to the AI safety community," Khanna told CNBC. "I believe they have been unfairly dismissed as science fiction and haven't been taken seriously enough. They need to be driving this debate, the AI safety community, not the executives of frontier labs."
Khanna described the measure as the most comprehensive AI safety legislation proposed to date.
Long Odds in a Divided Congress
The Human Control Over AI Act joins a crowded field of AI safety proposals on Capitol Hill. Among them is the bipartisan FRONTIER Act from Reps. Jay Obernolte, a California Republican, and Lori Trahan, a Massachusetts Democrat, which would place independent auditors in frontier labs and give the government authority to shut down models judged to pose catastrophic risk. A separate Sanders-Casar proposal to ban artificial superintelligence has drawn public backing from ten House Democrats, including Khanna himself.
The political reality, however, is sobering. No House bills are expected to receive a vote until after the midterm election, and the Senate is expected to leave Washington and not return until after voters go to the polls.
That timeline means the Human Control Over AI Act will function, at least for now, as a marker: a detailed blueprint of what the AI safety community would build if it held the pen — introduced at precisely the moment the people who build frontier AI are warning, on the record, that the technology is outrunning its controls.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
