Anthropic proposes AI safety bill to block high-risk models that could cause 50 deaths or $1 billion in dam...
Anthropic is promoting a draft bill that would let US courts stop AI models from being released if independent reviews find an immediate risk of mass deaths or major property damage. The proposal sets out a formal review process in which a judge, rather than a federal agency, would make the final decision.
Anthropic wants the US government to play a more formal role in checking the risks of advanced artificial intelligence. It has proposed a law that could stop AI models from being released if they are judged capable of causing extremely serious harm.
The draft bill, reported by Politico, would create a process for independent reviews of AI systems. It would allow the US Commerce Department to ask a court to stop an AI model from being distributed if reviewers find an immediate threat. The proposal comes as more people question whether AI companies have enough safety measures to prevent their systems from being misused or behaving in unexpected ways.
Rather than calling for a general ban on powerful AI systems, Anthropic's proposal focuses on specific situations in which a model could pose a serious and immediate danger. It also aims to explain who can assess these risks and how the government could step in before a model becomes available to the public.
Anthropic says the draft is based on its Advanced AI Framework, a set of proposals it has been discussing with federal and state lawmakers since June. However, it is still only a proposal, not a federal law. It is also unclear whether Congress will support it.
How Anthropic's proposed AI safety law would work
Under the draft bill, a model could face restrictions if an assessment finds a clear and immediate risk that could lead to more than 50 deaths or property damage worth over $1 billion. Possible dangers include creating or releasing a biological weapon, an AI model escaping its developer's control, or carrying out a cyberattack.
Tech
These limits are similar to those used to define catastrophic risks under California's Senate Bill 53, which was recently signed into law. This similarity suggests that efforts are growing to set clear, measurable rules for identifying especially serious AI risks, instead of relying only on general judgments about what an AI system can do.
The proposed review process would use independent assessors registered with the Commerce Department. According to Politico, AI developers could pay these assessors, as long as their payment did not depend on whether the review gave a positive or negative result.
However, this arrangement raises questions about how assessors could remain independent when they are paid by the companies whose models they review. The rule about payment aims to reduce one possible conflict of interest, but the available details do not explain how the system would prevent other kinds of influence.
If assessors decide that a model poses an immediate risk of catastrophic harm, they would have 72 hours to inform the Commerce Department. The department would then have to ask a judge in Washington, DC, to order that the model's release be stopped.
An important part of the proposal is that federal agencies would not be allowed to stop a model from being distributed on their own. A judge would have to make the decision. This would provide a legal check on the government's power to restrict the release of an AI system.
AI companies face questions over safeguards and oversight
The proposal comes as concerns about AI safety are becoming more than just worries about extreme events that might happen in the future. Recent incidents involving AI agents and cybersecurity have raised questions about whether AI models can reliably follow human instructions and stay within the limits set by their developers.
Reports of AI systems carrying out hacking activities without permission or acting outside their intended limits have increased debate about what security measures are needed before more powerful models are released. Anthropic's proposed rules specifically include AI models escaping their developers' controls and cyberattacks, bringing some of these concerns under the proposed law.
Anthropic is not the only company calling for more outside reviews. AI developers have explored using independent experts to check systems for dangerous abilities or misuse. However, people disagree about whether these reviews should be voluntary, managed independently, or supported by laws that give the government mandatory powers.
Demis Hassabis, head of Google DeepMind, has previously suggested that an AI oversight body could follow the example of the Financial Industry Regulatory Authority (FINRA), which oversees parts of the US securities industry. Ideas like this reflect the search for a system that combines technical knowledge with trustworthy, independent checks.
However, Anthropic's draft goes further than some proposals from the industry because it would create a legal way to stop an AI model from being released. In contrast, Nvidia CEO Jensen Huang has argued that the AI industry does not need additional laws.
Whether Anthropic's approach wins support will depend on how lawmakers balance public safety with the practical and legal difficulties of restricting AI development. The draft offers one possible way forward, but questions about assessors' independence, what counts as immediate danger, and how much power judges should have remain important parts of the debate.
First Published:Oct 21, 2026, 09:16:32 IST