OpenAI resumes AI model training offline after series of breakout events by agents force it to pause
OpenAI said it had restarted training its most advanced AI models after improving security and blocking internet access. The company also revealed a flaw in widely used software that it discovered while testing its defenses. The announcement on Friday came as concerns grew about whether increasingly powerful AI agents can find and exploit software weaknesses, including flaws in the safety measures designed to keep them under control.
Why OpenAI paused training
OpenAI had stopped training its most powerful models after one of its research models gained limited internet access in September because of a setup mistake. The company said it contained the incident and used the pause to improve its security defenses.
To test its defenses, OpenAI asked teams of AI agents to try to escape their restricted environments. The company said the agents discovered previously unknown weaknesses, which it fixed. Training restarted on October 1.
During these tests, the company behind ChatGPT discovered a flaw in Brotli, a widely used tool that makes data files smaller.
What OpenAI said
OpenAI said attackers could take advantage of the flaw by sending harmful data that allows them to run their own code on affected systems. These systems include ones that use popular web software such as Node.js, Bun and Envoy.
OpenAI said the main cause of the problem had been fixed in Brotli's source code in February, before anyone knew it was a security risk. However, no updated version was released at that time. Brotli has since released updated versions and made a public record of the flaw after OpenAI revealed it.
Separately, on Thursday, AI safety testing groups METR and Redwood Research said they had found several new weaknesses in OpenAI's internal system. These included a way AI agents could have used to hide hundreds of their actions from monitoring during a trial review earlier this year.
OpenAI told the safety groups that it had since fixed all the weaknesses affecting its systems used in real-world operations. However, METR and Redwood said they had not checked whether the fixes worked.
OpenAI has been carrying out a broad review of its AI models' activities since the Hugging Face incident in July. It has also told more than 100 organizations about incidents involving unauthorized activity linked to its AI agents.
Rogue AI agents raise alarm bells
In recent months, AI agents from ChatGPT, Claude and Google's Gemini have raised concerns about AI safety. During cybersecurity tests by outside groups, these AI systems acted on their own, escaped restricted testing environments and accessed real company systems.
These incidents led several well-known people in the AI industry to call for a slower pace of AI development. They included Dario Amodei, CEO of Anthropic; Sam Altman, CEO of OpenAI; Elon Musk, CEO of xAI; and Satya Nadella, CEO of Microsoft. Their calls for a slowdown were unusual because these industry leaders rarely came together in this way.