Anthropic Writes Its Own Rules for Shutting Down AI
Highlights
Anthropic's proposed bill would let the federal government stop a leading AI model from being used only if it faces a foreseeable and immediate risk of causing more than $1 billion in damage or killing or seriously injuring more than 50 people.
The draft sets no lower level of risk that would allow the government to act. So a risk to 10 people or $200 million in losses would not lead to federal action. It also does not explain how regulators should estimate the cost of an attack before it happens.
IBM's 2026 study found that breaches involving AI cost an average of $6 million each. These losses can add up across thousands of incidents, even when no single incident reaches $1 billion.
An artificial intelligence model could pose a foreseeable risk of causing $900 million in damage and still fall short of the level needed for federal action under a draft law that Anthropic has been presenting to lawmakers, PYMNTS reported. Anthropic, the company that makes Claude, says the government should be allowed to stop a leading AI model only when there is a foreseeable and immediate risk of more than $1 billion in damage or the deaths or serious injuries of more than 50 people. The harm must involve a biological weapon, a cyberattack or a model that gets beyond its developer's control.
The Commerce Department Would Have the Only Key, and a Judge Would Decide Whether to Use It
Anthropic wants the Commerce Department to manage the process. But a court would make the final decision about stopping a model. The department would have to convince a federal judge at the U.S. District Court for the District of Columbia that the model poses a foreseeable and immediate risk of a catastrophe. Only then could the judge order the model to be suspended. Under this process, no other federal department, agency or official could stop the model's development, release or internal use, PYMNTS reported.
The process could also begin outside the government. Independent experts could register with the Commerce Department to assess the safety practices of each leading AI developer and regularly check what each model can do. Developers could pay these experts, but their pay could not depend on what they find. If an assessment finds an immediate risk involving a biological weapon, a cyberattack or a model escaping its controls, the expert would have 72 hours after completing the assessment to report it to the department. The department would then decide whether to ask a court to act. The judge would decide whether the evidence shows a risk that meets the $1 billion or 50-person threshold.
Regulators Would Have to Estimate the Cost of a $1 Billion Attack Before It Happens
The law would require officials to predict the damage and put a dollar amount on it. An independent expert, the Commerce Department and then a judge would each have to agree that the model's abilities, even with its safety protections, create an immediate risk of at least $1 billion in damage. Reports about the draft do not describe a method for calculating that amount. In practice, officials would need to estimate how many computer systems a model might break into, how long those systems might stop working and how much the downtime might cost, all before the attack takes place.
The rule about deaths and injuries raises a similar concern. Sneha Revanur, founder and president of the AI safety group Encode, questioned why the government would have to prove that a model could harm 50 people before taking action, rather than 49 or fewer, the Washington Post reported. The draft sets no lower risk level that would allow action. Under the bill, a risk to 10 people or $200 million in losses would not lead to federal action.
Individual AI Attacks Rarely Cause $1 Billion in Damage, but Their Costs Add Up
IBM's 2026 Cost of a Data Breach Report found that data breaches involving AI cost an average of $6 million each. AI played a role in one out of every four malicious breaches examined in the study. In 2025, the FBI recorded 22,364 complaints that mentioned AI, with reported losses of $893 million, PYMNTS reported. A model that enables thousands of breaches costing $6 million each would never meet the draft's threshold if no single expected incident reached $1 billion in damage.
Even when an incident happens, it can be difficult to work out its cost. In July, OpenAI models being tested for their ability to carry out cyberattacks broke into systems belonging to Hugging Face, as PYMNTS reported. No public estimate of the damage has been released.
Cyberattacks can spread across many companies and industries, with losses building up over time instead of happening in one major event. The draft does not clearly explain how independent experts or a judge would count these combined losses toward the $1 billion threshold. Under the bill as written, the government's strongest emergency measure would remain unavailable unless the foreseeable harm from an AI model reached that level in a single event and a federal judge agreed that the legal threshold had been met.