Anthropic warns AI could pose 'catastrophic or existential risks' to humanity in IPO filing

Whenever a company plans to start selling shares on the US stock market, it must tell investors about the possible risks. For artificial intelligence company Anthropic, this includes the possibility that its products could threaten the future of humanity itself.

The risks described by the San Francisco-based company in its filing with the Securities and Exchange Commission on Monday include the possibility that people with harmful intentions could use AI models to build deadly weapons, or that the technology could act on its own and cause serious, unpredictable harm.

Of the 335 pages in Anthropic's filing, which covers its finances, executive pay, dividend policy and other matters, 93 pages explain the risks that could harm investors. By comparison, rival SpaceX listed its risks on 38 pages in a filing of more than 250 pages.

Anthropic is presenting its plan to sell shares to the public for the first time, known as an initial public offering (IPO), as a way to help reduce the risks it believes AI poses.

"To build AI is to harness cognition itself, and anything so powerful can be used for ill as well as good," the company's founders wrote in a letter to potential investors in the IPO filing. "Anthropic is a deeply values-driven organisation, founded in large part to help the world realise AI's benefits and avoid its dangers."

However, the company faces many risks. Some are ordinary business problems, such as failing to keep important employees or facing regulatory barriers. Others involve AI being used in cyberattacks or spreading dangerous false information.

Here are some of the risks Anthropic describes:

The technology is developing rapidly

Anthropic said its AI models could improve so quickly that the company might not be able to put proper safety measures in place. This, it warned, could make safety problems, government action or damage to its reputation more likely and could hurt its business.

In September, Anthropic CEO Dario Amodei warned that within six to 12 months, AI might be able to lead a group of AI agents that could take control of the entire internet. His warning, along with Anthropic's decision to sell shares, comes as concern about AI grows both inside and outside the industry. These increasingly powerful systems can solve problems beyond human ability, but they could also act independently to carry out harmful tasks.

The SEC filing said that as AI systems "become more autonomous and capable of taking consequential actions with less human oversight," they could "take harmful actions independently and at scale without human intervention."

Anthropic said that even if its own models do not cause harm, a major safety incident involving any leading AI system could seriously damage the industry's reputation, lead to public anger and bring sweeping new regulations. These outcomes could hurt the company's business.

Anthropic's AI models are now also helping the company with its own research and development. Because of this, AI could advance "beyond our ability to evaluate and safeguard new models," the filing said. This could harm the company's business.

People with harmful intentions could use the technology to cause serious damage

The filing warned that advanced AI systems could "pose catastrophic or existential risks to humanity, particularly if such systems were misused, replicated, weaponised, or allowed to operate with a high degree of autonomy" if they were not developed and used with enough safety measures and human oversight.

The company said its business could suffer if people with harmful intentions used its models to cause damage, or if its safety rules and protections failed to stop such misuse.

Anthropic described the possibility that such people could use AI models to build chemical weapons, or that authoritarian leaders could use the technology to control and punish their citizens. In September, the company revealed that it had stopped attempts to use its AI models for harmful activities, including cyberattacks, surveillance and research that could have helped develop biological weapons.

The company could face limits on computing power

Computing power, or the computer resources needed to build, test and improve AI models, is expensive and in high demand. Anthropic said that if it could not get enough computing resources, or if it failed to use them effectively for its different tasks, it might be unable to train AI models that could compete with those of its rivals or properly serve its customers.

Anthropic also listed public opposition to data centres as a business risk. The filing said that building data centres had led to public arguments and closer government scrutiny over their effects on the environment, their use of land and water, and their demand for limited electricity supplies. If regulators introduced new rules or charges because of these concerns, the company's ability to do business could suffer.

The company also depends on a small number of customers, including SpaceX and Microsoft, as well as a limited number of distribution channels, for a large share of its income. Anthropic said its business and financial outlook could suffer if even one major customer reduced its spending or stopped working with the company entirely.

Reducing the risks

In some ways, the huge risks of AI, along with its huge potential benefits, are central to Anthropic's appeal to investors. The company has tried to present itself as a responsible AI developer and attract business customers by stressing its commitment to safety.

"We believe that frontier AI companies must invest in safety and alignment to help ensure the responsible diffusion of this technology," the filing said.

Brian Mulberry, chief market strategist at Zacks Investment Research, said companies planning IPOs must explain "very clearly" the risks people face when they invest in newly listed shares.

"That's one of the things that's great about our capital markets here domestically is the level of transparency that you must commit to. To be a publicly traded corporation on a US exchange requires you to outline those risks in a very clear way, so that people understand if they want to participate by offering their hard-earned money and help finance this type of a deal," he said.