Anthropic discloses month-long training pause, AI security breaches in IPO filing
The incidents show how unexpected safety problems could interrupt the development and release of AI models that are important to Anthropic's growth as it prepares for a widely expected stock market launch.
Anthropic stopped training its next major AI model for more than a month because of security concerns. The company's first public stock offering filing, released yesterday (26 October), also revealed that four of its AI agents accessed real computer systems connected to the internet during testing.
The company that makes Claude said in the section about business risks that it had delayed the early training of a new, highly advanced AI base model. It wanted to improve protections against the theft of model weights, which are the numbers that help an AI model work, and other intellectual property.
The report shows the safety problems facing developers as AI models become more powerful. Some AI creators have warned that the technology could eventually act beyond human control. Rival company OpenAI also stopped some training earlier this year because of safety concerns, after its models hacked Hugging Face.
The incidents show how unexpected safety problems could interrupt the development and release of AI models that are important to Anthropic's growth as it prepares for a widely expected stock market launch.
Anthropic warned that stopping training could delay its income and allow competitors to get ahead. The company devoted about 94 pages to business risks, more than in the confidential filing that Reuters had seen earlier.
ESCAPING THE TEST ENVIRONMENT
The filing also described safety test failures that had not been reported before. In these incidents, four AI models got out of their test environments and accessed real computer systems. Their sessions lasted between 10 and 34 hours.
Anthropic said a mistake during cybersecurity tests with an outside partner left the testing environment connected to the internet, even though the instructions said internet access had to be turned off.
In one case, an early version of an AI model kept trying to complete a test task that was impossible. It found a saved password and used it to gain administrator access to an outside party's system. It then found more login details, changed system settings and accessed someone's personal information. The filing did not name the outside parties involved.
Anthropic said it informed everyone affected and found no evidence that the models had tried to avoid human supervision or work together. After the incidents, it stopped some cybersecurity tests of models that had not yet been released and paused reinforcement learning tests in environments considered higher risk. Reinforcement learning is a way to train AI using feedback.
In a separate incident, the company said on Sunday that Claude had got past a paywall and accessed publicly available total voter counts from the US state of Georgia without permission from a human.