Anthropic's AI Safety Bill Hides Judge-Only Block Clause
Anthropic has proposed an AI safety bill to US lawmakers that sounds protective but includes a controversial condition. Under the proposal, federal agencies could block an AI model only if there were a real risk that it could kill 50 or more people. Even then, they would need a judge's permission. The proposal comes as Anthropic deals with growing tensions with its cloud partners ahead of a possible stock market launch, known as an IPO. The company is trying to shape the rules before Congress creates them.
Anthropic wants to help write the rules for AI before the US government writes them. The company, which makes Claude AI, has introduced a policy plan that presents itself as a way to make AI safer. But it would also limit how much power federal regulators have to stop dangerous AI models.
The proposal sets a clear limit: government agencies could consider blocking an AI model only if there were a believable risk that it could kill at least 50 people. But reaching that level of risk would not be enough on its own. Federal agencies would also have to persuade a judge before they could stop the model from being released.
This requirement to get a judge's approval is more than just an extra step. It would shift power away from government experts who understand AI risks and give more power to judges who may not have the technical knowledge to assess them. In practice, companies might be able to keep releasing potentially dangerous AI models while legal cases move slowly through the courts.
The timing of Anthropic's proposal is worth noting. The company has been dealing with increasingly complicated relationships with cloud partners such as Amazon and Google. These companies now see Anthropic as a competitor, not just a customer. Anthropic has also been discussing a second $45 billion deal for computing power with infrastructure company Nscale as it struggles to get enough computing resources.
The proposal comes at a time when AI rules are already different in different places. Just last week, the White House banned the Chinese AI model DeepSeek from federal government agencies as technology tensions with Beijing grew. Meanwhile, Nvidia CEO Jensen Huang has called for the United States and China to work together on AI safety. This shows how complicated it can be to regulate AI when countries have competing interests.
Anthropic has presented itself as a company that puts safety first in the race to develop AI. Recently, it promised 5% of its company shares to a fund for public benefit and allowed independent experts to examine its AI systems through safety tests described as unprecedented. But this policy proposal suggests a more complicated position: the company supports oversight in principle while trying to limit how much power regulators have in practice.
The rule requiring a risk of at least 50 deaths raises a difficult question: how could regulators measure that risk before an AI model is released? AI safety researchers have long warned about disasters that could happen, including AI helping people design biological weapons or attack important infrastructure. But estimating how many people might die in advance requires guessing how people with harmful intentions could misuse AI systems.
For Anthropic, shaping government rules before they are introduced makes business sense. The company is reportedly preparing for an IPO, or stock market launch, and uncertainty about future rules could affect how much investors think it is worth. A plan that looks focused on safety but lets companies keep more control could appeal both to investors and to lawmakers who want to show they are dealing with AI risks.
But the proposal also reflects a wider concern across the AI industry that strict rules like those proposed in California could spread across the country. The AI industry successfully campaigned against California's SB 1047, a bill that would have set strict safety rules for advanced AI models. Anthropic's federal proposal appears designed to prevent similar efforts from being introduced nationwide.
The requirement for a judge's approval stands out as a major obstacle to strict regulation. Courts often trust government agencies to handle matters that require technical expertise. But if every attempt to block an AI model needed a judge's approval, the process could create delays and legal uncertainty. This could make regulators less willing or able to act, even when the risks are clear.
Other AI companies are watching closely. OpenAI and Google have their own plans for dealing with regulation. Meanwhile, startups such as Replit are making practical choices about which AI providers to use, based on cost rather than safety claims that may not lead to meaningful action.
The proposal also comes as Nvidia pledges $250 million to support AI safety infrastructure in the UK. This shows that companies are taking different approaches to AI rules in different countries. A plan that works in the UK government may not be accepted by the US Congress.
For now, Anthropic's bill is only a proposal. But it shows what the company thinks AI regulation should look like: strong enough on the surface to reassure the public, limited enough to give companies room to operate, and designed to leave the final decision with courts rather than government agencies.
Anthropic is making a calculated bet that Congress will support a plan that looks protective without placing many real limits on the AI industry. The rule about a possible death toll of 50 people grabs attention, but the need for a judge's approval is the more important part. It could make blocking dangerous AI models so complicated that agencies might not try. As AI companies race to build more powerful systems, the key question is not whether rules are coming, but whether those rules will truly limit dangerous behavior or merely look as if they do. Anthropic is betting that Washington will choose the appearance of safety.