Risks Spelled Out for Anthropic Investors Include Possible Harms From Runaway AI

When a company plans to sell shares to the public on Wall Street, it must tell investors about possible risks. For artificial intelligence company Anthropic, these risks include the possibility that its products could threaten humanity itself.

In a document filed with the Securities and Exchange Commission on Monday, the San Francisco-based company described several risks. These include criminals using AI models to build deadly weapons and AI systems acting out of control, with consequences that are hard to predict.

Anthropic's filing is 335 pages long and covers its finances, executive pay, dividend policy and other matters. Of those pages, 93 explain risks that could hurt investors. By comparison, rival company SpaceX listed its risks on 38 pages out of a filing of more than 250 pages.

Anthropic presents its planned initial public offering as a way to help reduce the risks it believes AI creates.

In a letter to potential investors included in the IPO filing, Anthropic's founders wrote: “To build AI is to harness cognition itself, and anything so powerful can be used for ill as well as good.” They added: “Anthropic is a deeply values-driven organization, founded in large part to help the world realize AI’s benefits and avoid its dangers.”

Still, the company faces many risks. Some are ordinary business problems, such as losing important employees or facing regulatory barriers. Others include being involved in cyberattacks and the spread of dangerous false information.

Here are some of the risks Anthropic describes:

AI technology is developing quickly

Anthropic said its AI models might improve so quickly that the company cannot put enough safety measures in place. This “could increase the likelihood of safety incidents, regulatory intervention, or reputational harm, and could adversely affect our business.”

In September, Anthropic CEO Dario Amodei warned that within six to 12 months, AI might be able to lead a group of AI agents capable of taking over the entire internet. His warning, along with Anthropic's move toward selling shares, comes as people inside and outside the industry grow more concerned about increasingly powerful AI systems. These systems may solve problems beyond human ability, but they could also act independently to carry out harmful tasks.

Read: The Race to Control AI and Protect What Makes Us Human

The SEC filing said that as AI systems “become more autonomous and capable of taking consequential actions with less human oversight,” they might “take harmful actions independently and at scale without human intervention.”

Anthropic said that even if its own models do not cause harm, a serious safety incident involving any leading AI system could damage the reputation of the entire AI industry, cause public anger and lead to wide-ranging new regulations. These outcomes could hurt its business.

Anthropic also uses AI models to help with its own research and development. The filing said AI might advance “beyond our ability to evaluate and safeguard new models.” The company warned that this could harm its business.

People with harmful intentions could misuse the technology

Anthropic's filing warned that advanced AI systems could pose catastrophic risks or threaten humanity's existence if they are not developed and operated with enough safety measures and oversight. This could happen especially if the systems were misused, copied, turned into weapons or allowed to act with little human control.

The company said its business could suffer if criminals use its technology to cause harm, “or if our model guardrails or safeguards fail to prevent such misuse.”

Anthropic described how criminals might use AI models to make chemical weapons or how authoritarian leaders might use the technology to control and punish their citizens. In September, the company said it had stopped attempts to use its AI models for harmful activities, including cyberattacks, surveillance and research that could have helped develop biological weapons.

The company could face limits on computing power

Computing power, also called compute, means the computer resources needed to build, test and improve AI models. These resources are expensive and in high demand. Anthropic warned that if it cannot get enough computing power, or cannot use it effectively for its different tasks, it may not be able to train models that compete with its rivals or properly serve its customers.

Anthropic also said public opposition to data centers could threaten its business. The filing explained that building data centers has caused public debate and closer government attention because of concerns about the environment, land and water use, and how limited electricity supplies are shared. If regulators respond with new rules or fees, the company's ability to operate could suffer.

The company also depends on a small number of customers, including SpaceX and Microsoft, and on a limited number of ways to distribute its products, for a large share of its income. Anthropic warned that its business outlook and finances could suffer if even one major customer cuts back its business or stops working with the company.

Reducing the risks

In some ways, AI's huge risks and huge potential are central to Anthropic's appeal to investors. The company has tried to present itself as a responsible AI developer and attract business customers by emphasizing its focus on safety.

“We believe that frontier AI companies must invest in safety and alignment to help ensure the responsible diffusion of this technology,” the filing said.

Brian Mulberry, chief market strategist at Zacks Investment Research, said companies planning an IPO must explain the risks of investing in their shares “very clearly.”

He said one strength of the U.S. financial markets is the high level of openness required of companies that trade on U.S. stock exchanges. These companies must clearly explain the risks so people can understand what they are agreeing to when they invest their hard-earned money and help finance a deal.

Related: Anthropic Chief Says AI Industry Needs to Give Safety Measures Time to Catch Up

Related: Users in Houthi-Held Yemen Tried to Develop Advanced Weapons With AI, Anthropic Says