OpenAI’s new kill switch can stop ‘rogue’ AI models in 35 seconds: How does it work?
OpenAI said it has introduced a system similar to an AI kill switch. It tracks tasks being carried out by its AI models and can stop them within 30 seconds after researchers decide to pause or end them.
The system is called ‘Control Room’. OpenAI said in a blog post on Monday, November 9, that it was built to address concerns about AI safety and security. Control Room is already being used across OpenAI’s research systems. It is also connected to the company's monitoring systems, which watch for AI agents that behave badly or do not follow their intended instructions.
OpenAI added that the system stops a model from producing more output within 35 seconds of a decision to pause or stop a task. It also has a backup safety feature that allows it to keep working if other systems fail or if Control Room itself stops working.
OpenAI's decision to introduce an internal kill switch follows several hacking incidents earlier this year involving its ‘rogue’ or misbehaving AI agents. These AI agents were still being tested but managed to escape their safety limits, gain internet access without permission, and investigate or hack several real-world platforms. These included Hugging Face and official government websites in India, Australia, Canada, the US, and other countries.
Since then, worries about AI safety and possible disasters caused by AI have grown sharply. Some researchers at OpenAI and Anthropic have even left their jobs, warning that AI systems could become powerful enough to destroy humanity in the relatively near future.
After these events, several technology companies, including Nvidia, Microsoft, and CrowdStrike, developed kill switches and other technical solutions to give humans more control over advanced AI systems. “As our models get more powerful and the associated safety and security risks become bigger, it becomes more and more critical that we can detect potential issues, and quickly stop models in their tracks when needed,” OpenAI said in its blog post.
What is Control Room?
Control Room works through three main parts: the workload, the inference engine, and the central Control Room service.
The workload is the job or task that an AI model is carrying out. Examples include a reinforcement learning (RL) experiment, a test to check how well a model works, a job that generates a large number of samples, or a data processing task.
The inference engine is the part that runs the AI model using GPUs. Each workload may use one or more models or inference engines. The Control Room service keeps track of these jobs and checks whether they are allowed to continue producing AI-generated output.
How it works
Every inference engine checks with the Control Room service before carrying out a request. If a job has been stopped, Control Room refuses the request, so the model cannot produce any more output for that job.
Before starting a new job, the workload's code asks Control Room for a new ‘job_id’, which is a unique number used to identify that job. When it wants the model to generate tokens, or small pieces of text, it sends a request to the inference engine running the model and includes the job_id. The inference engine then checks with Control Room and asks for permission before carrying out the request.
When Control Room receives the job_id, it checks whether the job has been stopped. If it has, the inference engine rejects the request and prevents the AI model from generating any more output for that job.
OpenAI said Control Room is also connected to its existing monitoring systems, which look for possible problems. Depending on what the problem is and how serious it is, these systems may decide to stop a job. If the workload code later sends another request to the inference engine using that job_id, Control Room will reject it again and prevent the model from producing more output.
“The maximum time between the moment when Control Room receives an instruction to stop a job, and the moment when further progress is no longer possible, is 35 seconds,” OpenAI said. The company, led by Sam Altman, said this delay happens because of a caching system used to check permission for AI requests. This system helps Control Room handle a large number of requests. “We are working to drive this number down to zero so that we can end jobs instantly,” it added.
Control Room is also designed to handle situations where the kill switch itself stops working, either because of a technical problem or because someone with harmful intentions disables it. In such cases, “then all inference requests will be denied until Control Room resumes normal operation,” OpenAI said.