OpenAI unveils a 'Control Room' safety feature that shuts down AI agents from going rogue in 35 seconds
OpenAI said Control Room will connect all research workloads that use AI models to produce results. This includes AI agents used to train and test models, process data, and other tasks. It will work with systems that monitor AI models for unsafe or unwanted behavior.
OpenAI has introduced Control Room, a feature that can stop AI systems from producing more output within 35 seconds if they raise safety or security concerns, the company said in a blog post.
The company explained why it needed Control Room: “As our models get more powerful and the associated safety and security risks become bigger, it becomes more and more critical that we can detect potential issues, and quickly stop models in their tracks when needed. To solve this, we built Control Room.”
Control Room is a central system that tracks the work being done by AI models across OpenAI's research systems.
OpenAI said Control Room will connect all research tasks that use AI models to produce results. These include AI agents used to train and test models, process data, and perform other tasks. It will also work with systems that monitor AI models for unsafe or unwanted behavior.
Control Room brings together several parts: the workload, the system that runs the AI model, and the central Control Room service.
A workload is a task being carried out, such as a reinforcement learning experiment, a test, a large-scale sampling task, or data processing. The inference engine is the part that runs an AI model and produces its output. The Control Room service tracks these tasks and decides whether they can continue producing output.
The blog post explained that inference engines must check with the Control Room service before handling a request. If a task has been stopped, the service refuses the request. This prevents the AI model from producing any more output.
OpenAI said it constantly checks for unauthorised or changed versions of inference engines that might try to avoid Control Room's safety checks.
“We additionally run continual agentic and programmatic sweeps to check that users are not accidentally or maliciously launching rogue engine implementations with Control Room unimplemented or disabled,” it noted.
OpenAI said it plans to improve and expand Control Room over the coming months. This work will happen alongside its efforts to monitor unsafe AI behavior and security breaches.