Rogue AI, cyberattacks and weapons: the risks Anthropic is warning investors about ahead of its IPO

Anthropic has warned investors about several risks as it prepares to sell shares on the stock market. These include AI systems acting dangerously on their own, cyberattacks and the possible use of AI to create dangerous weapons.

Of the 335 pages explaining Anthropic's finances, pay for its leaders, dividend policy and other matters, 93 pages cover "risk factors" — problems that could cause investors to lose money. By comparison, rival SpaceX listed its risks on 38 pages in a document of more than 250 pages.

As Anthropic tries to attract investors, it presents its planned initial public offering (IPO), which is when a company first sells shares to the public, as a way to help reduce the risks it believes AI creates.

In a letter to people considering investing, the company's founders wrote: "To build AI is to harness cognition itself, and anything so powerful can be used for ill as well as good." They added: "Anthropic is a deeply values-driven organisation, founded in large part to help the world realise AI's benefits and avoid its dangers."

However, the company faces many risks. Some are ordinary business problems, such as losing important staff or facing legal and regulatory barriers. Others include involvement in cyberattacks and the spread of dangerous false information.

Here are some of the risks Anthropic describes.

The technology is developing quickly

Anthropic said its AI models might improve so quickly that the company could fail to put proper safety measures in place. This, it warned, could make safety incidents, government action or damage to its reputation more likely, and could hurt its business.

In September, Anthropic CEO Dario Amodei warned that within six to 12 months, AI might be able to lead a large group of AI agents that could take over the entire internet. His warning, along with Anthropic's move to sell shares, comes as people inside and outside the AI industry grow more concerned about increasingly powerful systems. These systems can solve problems beyond human ability, but they might also act on their own to carry out harmful tasks.

As AI systems "become more autonomous and capable of taking consequential actions with less human oversight," the company's filing with the US Securities and Exchange Commission (SEC) said, they might "take harmful actions independently and at scale without human intervention."

Anthropic said that even if its own models did not cause harm, a serious safety incident involving any leading AI system could damage the reputation of the entire AI industry, cause public anger and lead to wide-ranging new rules. These changes could hurt its business.

Anthropic also uses AI models to help with its own research and development. The filing said AI could advance so quickly that the company might no longer be able to properly test new models or protect people from their risks. This could harm its business.

People with harmful intentions could use the technology to cause serious damage

The filing warned that advanced AI systems could pose catastrophic or even existential risks to humanity if they were not developed and used with enough safety measures and human oversight. These risks could be especially serious if the systems were misused, copied, turned into weapons or allowed to act with a high degree of independence.

The company said its business could suffer if people with harmful intentions used its AI models to cause damage, or if its safety rules and protections failed to stop that misuse.

Anthropic described the possibility that harmful users could use AI models to build chemical weapons, or that authoritarian leaders could use the technology to control and punish their citizens. In September, the company said it had blocked attempts to use its AI models for harmful activities, including cyberattacks, surveillance and research that could have helped develop biological weapons.

The company could face limits on computing power

Computing power, or the computer resources needed to build, test and improve AI models, is expensive and in high demand. Anthropic warned that if it could not get enough computing resources, or failed to use them well across its different tasks, it might be unable to train models that could compete with its rivals or properly serve its customers.

Anthropic also said that local communities' opposition to data centres could threaten its business. Building these centres has caused public debate and closer attention from regulators because of concerns about environmental damage, land and water use, and how limited electricity supplies are shared. The filing warned that new rules or fees introduced in response to these concerns could make it harder for the company to operate.

The company also depends on a small number of major customers, including SpaceX and Microsoft, and on a limited number of ways to distribute its products, for much of its income. Anthropic said its business and finances could suffer even if just one major customer reduced its spending or stopped working with the company altogether.

Reducing the risks

In some ways, the huge risks of AI, along with its huge potential benefits, are central to Anthropic's appeal to investors. The company has tried to present itself as a responsible AI developer and attract business customers by stressing its focus on safety.

"We believe that frontier AI companies must invest in safety and alignment to help ensure the responsible diffusion of this technology," the filing said. Safety and alignment mean making AI systems behave in ways that follow human goals and values.

Brian Mulberry, chief market strategist at Zacks Investment Research, said companies planning an IPO must explain the risks of investing in their shares "very clearly."

He said one strength of the US capital markets is the high level of openness required of companies that want to sell shares on a US stock exchange. They must clearly explain the risks so people can understand what they are getting into before deciding whether to invest their hard-earned money and help finance the company.