Anthropic pushes for rogue AI to be shut down only when 50 lives are threatened

Anthropic wants a law that would let the U.S. government shut down a dangerous AI model, but only if at least 50 people face an immediate risk of serious harm or death.

The AI company has been quietly asking members of Congress to support a draft bill. POLITICO obtained a copy of the bill. It would let states issue a "blocking order" if an AI model could cause at least $1 billion in damage to property.

The draft bill would also stop states from enforcing stronger safety laws. This could make it harder for courts to stop dangerous AI systems.

The bill would set a high standard for federal officials who want to act to prevent possible "catastrophic harm." For example, officials would have to prove that the AI threat was "imminent," meaning it was about to happen, and that there was no other way to prevent a disaster before they could step in.

AI critic Zvi Mowshowitz, a writer who focuses on AI, called the standard in the draft bill an "absurdly high bar."

The draft bill defines "catastrophic risk" as "a foreseeable and material risk" that a developer's development, storage, use or release of an advanced AI model would play a significant part in causing more than 50 people to die or suffer serious injuries, or in causing more than $1 billion in property damage in a single incident.

AI safety experts Nathan Calvin, Nat Purser and Nate Soares warned that a bill like this would weaken current protections against AI dangers and limit federal oversight, according to Novara Media.

Mowshowitz said the bill could leave people less safe than if lawmakers did "nothing." He argued that Anthropic seemed more worried about the government enforcing safety rules than making sure it had the power to hold the entire AI industry responsible.

Anthony Cimino, Anthropic's head of federal affairs, wrote on X that the draft bill's risk limits were "pulled directly from existing standards in California law."

He added that the bill would not free companies from legal responsibility or stop the government from using other available methods to protect the public.

Cimino also acknowledged that it might be "time to reevaluate thresholds to make sure the national standard is as strong as it can be."

Last month, former Anthropic employee Jacob Coxon warned that AI could wipe out humanity within a decade. His comments led other people working in AI to issue a series of warnings.

However, some observers have questioned these claims. Jeremy Kahn, editor of Fortune's AI coverage, said some people see them as a "cynical ploy ... intended to distract regulators and the public from the real and concrete risks that already exist."