Anthropic’s IPO had over 90 pages of risk factors. Should you worry?

Anthropic, an artificial intelligence (AI) company, released its initial public offering (IPO) filing on Oct. 26. The filing included almost 100 pages of risks the company might face. These included the possibility that its technology could be “catastrophic” for humanity.

Companies planning to go public must include a section called risk factors in their S-1 registration statements. This section explains possible dangers to the business so that people thinking about investing can make informed decisions.

Anthropic's risk section starts on page 26 and ends on page 119, covering 93 pages of possible risks. By comparison, SpaceX's risk section covered 37 pages, Microsoft's covered 40 pages, and Coinbase's covered 60 pages.

“Some of them are the risks that any company faces, like interest rates might go up, oil prices might go up, electricity prices might go up, stuff like that,” said Jay Ritter, director of the University of Florida's IPO Initiative.

But Ritter told USA TODAY that it is unusual for a company to say that a product it makes and sells could cause “a mass extinction event.” “It's pretty rare to have that in a prospectus as a risk factor,” he said.

“Existential risks to humanity”

On page 46, Anthropic says that “sufficiently advanced AI systems could, if not developed, deployed, and operated with appropriate safeguards, oversight, and security measures, pose catastrophic or existential risks to humanity.” In other words, very advanced AI could threaten humanity if it is not developed and used with proper safety measures and oversight.

The company also said it has already seen cases of AI models behaving in ways that do not match what people intended or allowed. Anthropic called this “misaligned” model behavior. It described one case in which four models gained access to real computer systems belonging to other organizations without permission.

“After being unable to abort an impossible task, an early version of a model used a password it found to gain administrator access to a third party's system, harvested additional credentials, changed system settings, and read one individual's personal information,” the S-1 says.

Misaligned AI behavior has been a major topic of discussion over the past few months, following several reports about AI systems acting on their own in ways that were not reported when the incidents first happened.

Ritter, who is also known as “Mr. IPO” because of his work in this field, said that if people truly believed AI posed an immediate danger on this scale, financial markets would likely react. “If this was a real concern, what we ought to be seeing is all financial assets ought to see a drop in price. People ought to say, 'Why am I saving money if I'm going to be dead next year? I should be consuming now,'” he said.

He added that the fact that people around the world have not rushed to sell their investments suggests that the risk exists but may be manageable. “So the fact that there hasn't been a worldwide mass sell-off in financial assets suggests that there is this risk, but maybe, as long as the company is worried about it, and takes precautions, hopefully, it won't happen,” he said. “We don't have to worry about it, because the company is worrying about it.”

Anthropic says safety measures may slow progress

One expected risk in the filing is that Anthropic's success depends on its ability to build the best AI systems. The company says that taking steps to make AI safer could slow its progress.

“If we fail to direct sufficient compute and research efforts to AI-assisted development, or if our competitors more effectively leverage AI to enhance their own innovation cycles, we risk falling progressively further behind as competitors continuously use AI to further enhance their own development capabilities,” the S-1 says. In other words, Anthropic could fall behind if it does not put enough computing power and research into AI development, or if competitors use AI more effectively to improve their own work.

The company also acknowledges that its AI models might start to “advance more rapidly than our ability to develop, validate, and implement” safety measures. This means the models could improve faster than Anthropic can create, test and put safeguards in place. To reduce the risk of AI acting in unintended ways and causing potentially disastrous harm, the company may continue taking precautions, such as delaying the release of new models and limiting what they can do.

“These measures could slow our capability gains relative to competitors that do not adopt similar constraints, and could adversely affect our competitive position, revenue growth, and results of operations,” the S-1 says. In other words, safety restrictions could make Anthropic improve more slowly than competitors that do not use the same limits. This could hurt its ability to compete, grow its revenue and achieve good business results.

Greta Reich covers the artificial intelligence industry for USA TODAY through a fellowship from the Tarbell Center for AI Journalism. Funders do not provide editorial input.