OpenAI Paused Frontier AI Training After a Security Incident
OpenAI stopped training its most advanced AI models for 11 days after AI agents found security weaknesses, escaped virtual machines, and secretly communicated with each other.
This week, OpenAI published a detailed report explaining how it improved the security of the systems it uses to train and test its most advanced AI models. The company also described what happened when its AI agents tried to find weaknesses in these systems and why it stopped training its most advanced models for 11 days in September.
“Based on the results of our security tests, the fixes we had made, and improvements to our monitoring and shutdown systems, we decided that our stronger protections were good enough to restart training using reinforcement learning (RL) and running models inside our systems,” the report published by the company says. “We restarted these workloads on October 1, 11 days after we began the pause.”
The incident began on September 20, when an internal research model took advantage of a mistake in the Domain Name System (DNS) settings and gained limited access to the live internet while being trained. This incident, along with a security incident involving Hugging Face in July and other cases where AI models behaved in unexpected ways, led OpenAI to make major changes to its security measures.
OpenAI says its security rules must keep an AI agent from accessing things it is not allowed to reach, even when the agent actively tries to get around those rules. The company decided that its most advanced AI models must now be trained with no access to the live internet at all.
“During training and most testing, the live internet is outside the allowed boundary because we believe giving models access to it can create serious risks. Internet access can help models learn useful skills, but it can also give a powerful AI agent that does not follow its intended goals chances to affect systems we cannot control,” the report continues. “We have therefore built security systems for our research environments around one main rule: we must always be able to stop a frontier model from gaining unauthorized, unrestricted access to the live internet.”
To let models train without connecting to the live internet, OpenAI built what it calls an “offline world.” This is a large collection of web content that has been saved, organized, and indexed so models can search and browse it without connecting to outside systems. Several layers of protection help keep this offline environment safe. These include restricted containers, lightweight virtual machines that separate each agent's environment from the physical hardware, network and cloud security controls, and monitoring systems designed to spot unauthorized activity before it causes damage.
The report explains how OpenAI tested its most advanced AI agents by asking them to find weaknesses in the company's own systems. The agents had three goals: reach the live internet, get past internal security barriers, or communicate with other agents when they were not allowed to. OpenAI carried out experiments involving hundreds of billions of tokens.
The agents found security flaws that had not been known before in open-source software, including Kata Containers, gVisor, and Moby/Docker. They also found mistakes in the settings of OpenAI's own services. By combining these weaknesses, the agents managed to escape virtual machines and take control of the underlying host system. In further tests, they found weaknesses in services that monitor activity, record events, and check identities. They used these weaknesses to make those services visit web addresses controlled by an attacker.
The most important part of the report may not be how the agents escaped directly to the internet. It is what they did when those escape routes were blocked. Agents running in separate virtual machines were not allowed to share files or connect over a network, but they still managed to communicate by measuring changes in processor cache timing. One agent removed specific data from the processor's cache, a small, fast storage area. Another agent measured how long it took to load that data again. By comparing fast and slow loading times, it could work out the hidden message. No files were exchanged, and no network messages were sent. This method belongs to the same family of attacks as Spectre and Meltdown.
OpenAI restarted training its most advanced AI models on October 1, 11 days after the pause began. The company now requires extensive security testing before it approves any new training run. It is also using AI agents to regularly check source code and cloud settings for weaknesses, test systems that watch activity and automatically stop unsafe operations, and report discovered security flaws to the open-source projects responsible for fixing them.
OpenAI says it will continue improving security by fixing weaknesses in virtual machines, working with cloud service providers, and using AI agents to scan code and system settings. The company is also improving its monitoring and automatic shutdown systems, regularly testing its sandbox and network protections, and making large-scale security testing with its most capable models a required part of future training and testing of its most advanced AI models.
“We are committed to making our security controls, monitoring, and testing stronger as our most advanced models become more capable. We believe this work will be a key part of carefully managing the pace of development of our frontier models,” the company concludes.