Dario Amodei's AI Safety Warning Is Colliding With Anthropic's IPO Timeline

Anthropic CEO Dario Amodei has warned several times in recent months that artificial intelligence (AI) models may advance faster than the safety measures needed to control them. Even so, Anthropic is reportedly still planning to sell its shares to the public for the first time in November. This is called an initial public offering (IPO).

On Oct. 14, New York City Comptroller Mark Levine urged Anthropic to delay its IPO until it addresses concerns about AI safety and how the company is run. Bloomberg reported that Anthropic later contacted Levine, but there was no reported change to the IPO schedule.

Safety concerns have already interrupted AI development

In an essay published in September, Amodei warned that AI could improve itself in a repeated cycle. This is known as "recursive self-improvement." It means AI systems could change their own programming and keep developing on their own. He is concerned that this could create powerful AI systems that their developers can no longer control or fully understand. In his view, current safety measures will not be enough to keep up.

Amodei has called for independent safety checks and more cooperation between AI companies and governments. Shane Legg, a co-founder of Google DeepMind, which is owned by Alphabet, has also warned that AI could advance faster than the safety measures designed to control it.

These concerns have already affected Anthropic's work. On Aug. 31, Anthropic said it had paused some high-risk training activities involving AI models that had not yet been released. The pauses lasted several weeks. Most affected work involved reinforcement learning, a method in which AI learns through feedback. This work restarted after Anthropic added more monitoring.

On Oct. 9, Anthropic reported more incidents involving AI systems that did not behave as intended and problems related to security. The company also announced plans to keep limits on live internet access in place for all internal model tests until its monitoring systems can reliably detect similar incidents. It is still unclear how these limits will affect the company's finances.

Anthropic's AI safety decisions

Anthropic is taking steps to improve independent safety checks, but outside experts cannot stop the company from releasing an AI model. On Oct. 15, Anthropic announced that six researchers from the nonprofit group METR had been given access similar to that of its own safety teams. METR can publish its findings without Anthropic's approval, but it cannot decide whether an AI model should be released.

Anthropic's Long-Term Benefit Trust chooses most of the company's board members. Its goal is to help balance financial interests with AI safety. Reuters has also reported that a draft plan for Anthropic's IPO would give its seven co-founders 50.1% of the voting power on important company matters through a new organization called Founder LLC. Anthropic's draft prospectus also warned that this setup could lead to decisions that do not put financial results or shareholder value first. As a result, if safety-related delays hurt Anthropic's financial performance, public shareholders would bear the costs but have limited power over major company decisions.

Anthropic also wants the U.S. government to play a more direct role in setting AI safety rules. Reuters reported that the company has been presenting Congress with a draft law that would allow the Department of Commerce and federal courts to block certain AI models if they were judged to be dangerous. If passed, the law could delay the release of AI models for commercial use. It could also affect the growth expectations behind Anthropic's planned IPO valuation.

Safety decisions could affect Anthropic's IPO valuation

Anthropic's revenue grew 45% from the previous quarter to $16.8 billion in the third quarter. However, some major investors invited to the company's investor day in San Francisco last week were reportedly discussing whether an IPO valuation of about $1.5 trillion would make more sense than the company's $2 trillion target. These investors are concerned that revenue growth is slowing and the cost of building and running AI infrastructure is rising. There is no clear evidence that safety-related limits caused this possible change in valuation.

So far, Amodei's warnings about AI dangers have not changed Anthropic's reported IPO schedule. But the company still needs to show that it can put the right safety measures in place while also delivering the growth and profits needed to support its ambitious valuation.