Anthropic’s weak AI bill undermines its ‘safety first’ approach

Anthropic has always presented itself as the AI safety company. It says it is willing to take costly steps and support strict AI rules, even when doing so hurts its own business. For years, a key part of its plan for making change has been to become a leader in AI, earn a place in discussions with officials in Washington, and use that position to push for rules strong enough to address the most serious risks from AI.

A leaked draft of a federal AI bill that Anthropic has been sharing with lawmakers was published by Politico on Friday. It makes Anthropic’s claim that safety comes first hard to believe. AI safety advocates have widely criticized the bill. It is much weaker than the Obernolte/Trahan FRONTIER Act and could even make it harder under the law to stop a dangerous AI model from being released than it is today. The proposal looks as if Anthropic’s business interests have taken priority over its stated safety concerns. And at a time when there seems to be a real chance to shape AI policy, the bill could close the door to stronger rules in the future, even as the company itself says AI risks are growing.

Three parts of the bill, which you can read in full here, stand out. Like the FRONTIER Act, it requires outside experts to assess AI companies’ models and practices for risks that could lead to disasters. But unlike FRONTIER, it sets no standards for these evaluators and does not require them to get a license. Evaluators can simply certify themselves. As LawAI’s Charlie Bullock points out, this means “auditors will inevitably compete with each other to offer … the least bothersome audits available at the lowest possible price point.” In other words, companies that check AI systems may compete to offer the easiest and cheapest checks.

The bill also makes it very difficult to stop a dangerous AI model. Under Anthropic’s proposal, the Commerce Department would be responsible for seeking to block a model. To do so, it must ask a federal court for an order and provide detailed reasons why the model is dangerous. Its case must be separate from anything an independent evaluator has said. The court must make every decision “without deference” to the Commerce Department or the independent evaluators. It must also “consider the harm the [blocking] order would cause to the frontier developer, to users, and to the public.” Nathan Calvin of Encode, who said the bill made him “viscerally angry,” points out that the court is not told to consider the harm that could come to the public if the model is not blocked.

Understand AI – and what to do about it

The bill also raises the long-running debate over whether federal law should prevent states from making their own rules. In most ways, Anthropic’s bill treats this issue much like FRONTIER. It would override state laws aimed at regulating “frontier AI,” including California’s SB 53, New York’s RAISE and Illinois’ SB 315. But it clearly leaves out laws on child safety and some other areas. Unlike FRONTIER, however, Anthropic’s proposal would also prevent state lawmakers or courts from suspending an AI model because it poses a catastrophic risk. According to Calvin, this means the bill “likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models.”

The bill is also unusually inflexible. It would never let the Commerce Department set standards for AI evaluators or define what makes an AI model “safe.” And if state AI laws are overridden, stronger rules would have to come from Congress alone. This is hard to defend given Anthropic’s own views about how quickly AI is advancing. Company co-founder Jack Clark recently said he thinks AI might achieve full recursive self-improvement next year, meaning it could improve itself again and again. As Nat Purser of AVERI says, “if you’re going to preempt many state safety laws, the least you can do is create a federal regulatory regime that can evolve in time.” In other words, if a federal law replaces many state safety laws, the federal rules should at least be able to change as risks develop.

The bill is not entirely bad. It includes some important improvements over FRONTIER, including good rules for how AI companies should protect their systems and ways to deal with failures to contain AI and risks from AI doing research and development automatically. But these gains are outweighed by two major setbacks. The timing makes this especially troubling. FRONTIER was introduced in July, after a draft was shared in June, when AI safety was getting much less political attention. After a summer of incidents involving AI systems behaving in unexpected or dangerous ways and warnings from employees about possible disasters, the situation has changed. Stronger state laws are on the way. This is the time for stronger federal bills, not weaker ones.

It is not hard to see why the bill may have taken this shape. It looks like the work of a company that has been targeted by an unpredictable executive branch of the government and is now focused on protecting itself. As writer Zvi Mowshowitz says, “Anthropic is more afraid that the government might enforce rules against Anthropic, than they are worried about the ability of the government to enforce rules against everyone.”

There is some reason to hope the bill will improve. After the leaked draft drew strong criticism, Anthropic’s head of federal affairs, Anthony Cimino, said its language overriding state laws was “not precise enough” and would be revised. He also suggested that future policy proposals would address some of the bill’s other problems.

But a company that says it is committed to AI safety should not need public criticism to produce a good bill. The strong reaction from across the AI safety community suggests that many people feel betrayed. They expected Anthropic to stand by their shared goal of making AI safer, but now feel that it has done what other companies have done and put business interests first.