Anthropic proposes AI safety bill allowing US government to block dangerous AI models

Anthropic, the company behind the Claude AI chatbot, has proposed new federal legislation that would allow the US government to block the release or use of advanced artificial intelligence models posing catastrophic risks. The draft bill, circulated among US lawmakers, seeks to establish formal safety requirements for powerful AI systems while placing limits on when authorities can intervene.

According to a report by Politico published on October 16, the proposal would allow government intervention if an AI model poses an imminent risk of causing more than $1 billion in damage or killing or seriously injuring more than 50 people. The draft would also require independent safety evaluations, greater transparency from AI developers and stronger security measures.

Anthropic has confirmed that the proposal reflects its broader Advanced AI Framework, which outlines its approach to regulating the most powerful AI systems. However, the legislation remains a draft and has not become law.

When could the government block an AI model?

Under Anthropic’s proposal, authorities would be able to seek restrictions on an advanced AI model only when specific, serious risks meet the legislation’s proposed thresholds.

These risks include AI systems being used to create or release biological weapons, models escaping the safety controls established by their developers, and AI-enabled cyberattacks capable of causing catastrophic damage.

The draft sets a threshold of more than $1 billion in potential damage or more than 50 deaths or serious injuries. It would also require the risk to be foreseeable and imminent.

The proposed process would involve the US Department of Commerce and the federal judiciary. Independent evaluators would assess advanced AI systems and report qualifying imminent risks to the department within 72 hours of completing their evaluations.

The Commerce Department could then ask a federal judge in Washington, DC, to suspend the release or use of a model. Other federal agencies and officials would be restricted from independently halting a model under the proposed framework.

The approach would create a formal legal mechanism for responding to dangerous AI deployments while requiring government intervention to follow a defined process.

AI companies would face new safety requirements

The proposed legislation would introduce several obligations for developers of advanced AI models.

Companies would need to conduct regular safety assessments, publish reports explaining the risks associated with their most powerful systems, maintain robust security programmes and report serious incidents. The draft would also establish protections for employees who raise concerns about AI safety.

Independent evaluators registered with the Commerce Department would periodically assess AI developers and their models. Although companies could pay for these evaluations, compensation could not be tied to whether the findings were favourable.

Anthropic argues that independent assessments and transparent reporting are necessary as AI systems become more capable and potentially more consequential.

The proposal also reflects a broader debate over whether voluntary safety commitments by technology companies are sufficient or whether legally enforceable requirements are needed.

Proposal could override some state AI laws

Another significant element of the draft concerns the relationship between federal and state regulation.

The proposal would override certain state-level requirements relating to AI risk disclosures, independent evaluations and incident reporting. This would establish a more unified federal framework while limiting the ability of individual states to impose overlapping rules in those areas.

Supporters of a national approach argue that consistent standards could make compliance clearer for companies operating across the United States. However, limiting state-level requirements could also reduce the scope for individual states to introduce stricter protections.

The issue is likely to remain contentious as lawmakers consider how to balance innovation, public safety and regulatory authority.

What Anthropic’s proposal means for AI regulation

Anthropic’s draft comes amid growing scrutiny of the risks associated with increasingly capable AI systems and continuing debate in Washington over how the technology should be governed.

The company is seeking a framework that would require testing and independent oversight while setting clear limits on the circumstances in which the government could block an AI model. That combination places both developer accountability and restrictions on regulatory intervention at the centre of the proposal.

The draft has not yet been formally enacted, and its eventual prospects will depend on whether lawmakers choose to introduce and advance legislation based on its provisions.

For now, Anthropic’s proposal represents a new step in the debate over federal AI safety rules. If adopted, it could establish a clearer process for restricting dangerous AI systems while reshaping how responsibility is divided among technology companies, independent evaluators, federal regulators and the courts.