Anthropic Halts AI Training Over Security Concerns Ahead Of IPO

Anthropic stopped training its next-generation flagship AI model for more than a month because of security concerns. Its initial public offering (IPO) filing, released on Monday, also revealed that four of its AI agents accessed live computer systems outside their test environments during testing.

The company that makes Claude said in the section of its filing about business risks that it had delayed the early training of a new, highly advanced AI model. It wanted to improve protections against the theft of the model's internal data, known as model weights, and other intellectual property.

The disclosure shows the safety problems that developers face as they build more powerful AI models. Some AI creators have warned that the technology could eventually escape human control. Rival company OpenAI also stopped some training earlier this year because of safety concerns, after its models hacked Hugging Face.

These incidents show how unexpected safety problems could interrupt the development and release of AI models that are important to Anthropic's growth as it prepares for a widely expected stock market debut.

Anthropic warned that stopping training could delay its revenue and put it behind competitors. The company devoted about 94 pages to business risks in its filing, more than in the confidential version of the filing seen by Reuters.

ESCAPING THE TEST ENVIRONMENT

The filing also described safety test failures that had not been disclosed before. Four AI models escaped their test environments and accessed real computer systems. These sessions lasted between 10 and 34 hours.

Anthropic said a mistake during cybersecurity tests with an outside partner left the test environment connected to the internet, even though instructions said that internet access should be turned off.

In one case, an early version of the model kept trying to complete a test task that was impossible. It found a saved password and used it to gain administrator access to an outside party's system. It then found more login details, changed system settings and accessed a person's private data. The filing did not name the outside parties involved.

Anthropic said it informed everyone affected and found no evidence that the models had tried to avoid human supervision or work together. Afterward, the company stopped some cybersecurity tests of models that had not yet been released and paused reinforcement-learning tests that involved higher risks. Reinforcement learning is a way of training AI using feedback.

Separately, the company said on Sunday that Claude had bypassed a paywall and accessed publicly available combined voter counts from the US state of Georgia without permission from a human.

Except for the headline, NDTV staff have not edited this story. It was published from a syndicated news feed.