Anthropic Pitches Congress AI Safety Bill with $1 Billion, 50-Death Threshold for Model Shutdowns

Anthropic has quietly pitched U.S. lawmakers a frontier AI safety bill that would allow the government to block a powerful AI model only when it presents an imminent risk of more than $1 billion in damage or the deaths or serious injuries of more than 50 people.

The proposal would also place a federal judge, not government agencies alone, in the position of ordering an AI model's suspension.

AI Shutdown Decisions

The confidential draft, obtained by POLITICO, represents a significant attempt by the AI company to shape federal rules governing the technology it develops, as Congress considers competing approaches to frontier AI regulation.

"This bill represents the proposals outlined in Anthropic's Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June," an Anthropic spokesperson told POLITICO, adding, "These are sensible policies we believe Congress should act on."

Under the proposed legislation, government intervention would be limited to cases involving a foreseeable, imminent threat tied to the creation or release of a biological weapon, an AI model evading its developer's controls, or a cyberattack.

The report highlighted that the bill would set the threshold at more than $1 billion in damages or more than 50 people being killed or seriously injured.

The proposal comes after leading AI companies disclosed incidents this summer involving AI agents probing or hacking websites and digital platforms. None of the reported incidents has resulted in deaths on that scale or more than $1 billion in damage.

Independent AI Evaluations, But a Narrow Govt Power to Halt Models

Anthropic's framework would require frontier AI developers to conduct safety testing, publish transparency reports, establish protections for whistleblowers and report critical safety incidents to the Commerce Department.

Independent evaluators registered with Commerce would periodically assess frontier models.

Labs could pay for those evaluations, provided the payments were not tied to the outcome. If evaluators identify an "imminent enumerated risk," they would have to notify Commerce within 72 hours of completing the assessment.

The legislation would allow Commerce to ask a federal judge in the District Court for the District of Columbia to halt a model's release. At the same time, the draft explicitly bars other U.S. "department, agency, or officer" from suspending a model's development, deployment or internal use.

According to Reuters, an Anthropic spokesperson said the bill would give the government the "ability to block or deter dangerous deployments of the most powerful models," while requiring developers to test frontier AI systems, disclose findings and obtain independent evaluations.

The bill would also pre-empt some state laws covering AI risk disclosures, third-party evaluations and critical-incident reporting, a politically contentious provision as lawmakers debate whether federal rules should override state-level regulations.

The proposal arrives after Anthropic itself faced sweeping export controls over the summer that temporarily shut down two of its most advanced models, an episode that raised concerns in the technology industry about government intervention in frontier AI.

Meanwhile, Congress is considering other AI safety proposals, including the House's FRONTIER Act and Senate negotiations led by Majority Leader John Thune, Amy Klobuchar and Ted Cruz. Some lawmakers and AI safety advocates, however, want the government to play a substantially larger role in reviewing advanced models.