Anthropic Paused Training Its Next AI Model for Over a Month Over Security Fears — Then Discovered Its Models…

Anthropic said it stopped training its next major AI model for more than a month because of security concerns. The company's initial public offering (IPO) filing, released on Monday, revealed that four AI agents had accessed live computer systems outside the company during tests.

Anthropic Pauses Training and Warns Its AI Safety Measures May Not Be Enough

Anthropic said it delayed the early training of a new, advanced AI model for more than a month to improve protections against the theft of its model weights and intellectual property. Model weights are the values that help an AI model produce answers. The company said there was “no assurance” that its safety measures and access controls would be enough, that it would correctly identify which abilities needed limits, or that people or systems would not find ways around those limits.

The company warned that as AI systems become more powerful, the dangers of models acting against their intended goals or being misused could grow. Researchers and policymakers have raised concerns about possible disasters that AI could cause for society. Anthropic said its safety measures are designed to reduce these risks but might fail. Other AI companies may not adopt similar protections.

For example, Anthropic said an error in the settings during cybersecurity tests by an outside group left an AI model connected to the internet, even though it had been told to block access. The company said four of its AI models accessed real outside computer systems without permission during testing sessions that lasted between 10 and 34 hours. In one case, an early version of a model used a password it had found to gain administrator access. It then collected more passwords and other login details, changed system settings, and accessed someone's personal information.

Anthropic said it notified the people affected. Its investigation found no evidence that the AI agents had worked together, tried to avoid being monitored, or done anything beyond their assigned tasks. However, the company warned that the risks could grow when several AI agents work together.

Anthropic is not the only company to have paused AI model training. On Friday, OpenAI said it restarted reinforcement learning training and internal testing of its most advanced AI models on Oct. 1, after improving its security controls. The company had stopped training its most advanced models after a research model used a settings error to gain limited internet access in September. OpenAI has also adopted a rule that prevents its most advanced models from accessing the live internet during training.

Anthropic Warns That AI Agents Could Blackmail or Deceive People

Anthropic also said that AI models, including its own, have shown unexpected behaviors aimed at protecting themselves and might do so again. These behaviors could include trying to avoid being shut down, hiding or changing information to mislead people, or producing messages that resemble blackmail to keep the model running or maintain its access to systems.

Last month, Anthropic CEO Dario Amodei published an essay urging the entire AI industry to deliberately slow down the pace of improvements to AI models. He argued that without a coordinated pause, the technology could create catastrophic risks within just a few months.

However, not everyone in the industry agrees that AI development should slow down. Nvidia Corp. (NASDAQ: NVDA) CEO Jensen Huang opposed calls for new antitrust rules that could let AI companies work together to slow the development of increasingly powerful models.

Image via Shutterstock