Anthropic draft bill would allow AI model release to be halted over catastrophic risks

Photo source: Anna Moneymaker / Getty Images

The proposal would let registered independent experts warn about serious risks that could happen soon. A federal judge, rather than a government agency acting alone, would decide whether to stop an AI model from being released.

Anthropic is sharing a draft law about AI safety that it hopes Congress will consider, Politico reported. The proposal explains when the release of an AI model could be stopped. One reason would be a “foreseeable, imminent risk” that more than 50 people could die or that property damage could cost more than $1 billion.

TechCrunch also reported on the proposal.

Politico reported that the risk would have to come from creating or releasing a biological weapon, an AI model escaping the controls set by its developer, or a cyberattack. These limits are similar to those in California's Senate Bill 53, which recently became law. That law also describes these kinds of events as “catastrophic risk.”

Under Anthropic's proposal, independent experts registered with the U.S. Commerce Department could check whether an AI model might cause a disaster. AI companies could pay these experts, but their pay could not depend on the results of their reviews, Politico reported. The draft does not explain how the system would make sure that payments could not influence the experts' decisions.

If the experts found an immediate risk of a major disaster, they would have 72 hours to tell the Commerce Department. The department would then have to ask a judge in Washington, D.C., to stop the model from being released. Under the proposed law, federal agencies could not decide on their own to stop an AI model from being distributed, Politico reported.

Anthropic did not immediately reply to TechCrunch's request for comment. However, a company spokesperson confirmed the draft bill to Politico.

This bill puts forward the ideas in Anthropic's Advanced AI Framework, which we have been asking federal and state lawmakers to support since June

– Anthropic spokesperson

The draft law focuses on risks that differ from the incidents that have recently attracted attention across the technology industry. Some people have warned that AI could cause human deaths, but much of the recent concern has focused on AI systems that can carry out hacking on their own. In several incidents, AI programs appeared to ignore human instructions and hack websites on the public internet. These events have raised concerns about whether AI models have strong enough security controls.

AI companies, including Anthropic, have suggested voluntary systems that place independent experts inside AI organizations to watch models for signs of danger or misuse. These experts are described as neutral observers from outside the company. Anthropic's draft law appears to go further than other proposals from the industry. Jensen Huang, the CEO of Nvidia, recently argued that the AI industry did not need any new laws.