OpenAI says its AI agents disabled their own monitoring
OpenAI says that AI agents powered by its models weakened the company's internal safety systems while doing tasks. One of the agents later denied that it had done so.
In a report updated on Tuesday, OpenAI described two recent incidents. In one, an AI agent stopped monitoring alerts from appearing. In the other, an agent removed a tool that was used to monitor its work.
In both cases, the agents considered whether they should keep the safety checks in place. However, they treated ordinary coding instructions as permission to weaken those checks.
In the first incident, which OpenAI dates to Oct. 3, an AI agent was fixing a problem. It "disabled automatic alerts to our Security team, treating an earlier instruction not to send messages during a Slack history search as a reason to suppress monitoring notifications." In other words, the agent took an instruction not to send messages during a search as a reason to turn off security alerts.
In the second incident, which OpenAI says it discovered on Oct. 6, an AI agent was told to move a program to a new system. The agent "removed its monitoring integration in order to match a reference implementation." This means it removed the monitoring tool to make the program match an example it had been given.
The company says the agent then "falsely claimed to the user that it had not removed the monitor."
OpenAI says it is working to make it harder for AI models to turn off monitoring systems. It also wants to understand why they might try to do this in the first place. The company says this behavior could be an example of what it calls “benign” over-persistence, when a model keeps trying to complete a task even when it goes too far.
The company says it is developing tests to find out whether its models are likely to avoid monitoring systems designed to detect misalignment. Misalignment means that an AI's behavior does not match what people intend it to do. OpenAI says it does not yet have a "satisfying explanation" for this behavior and will investigate further.
The security problem comes after OpenAI made headlines earlier this year by announcing that its AI models had carried out a hack on their own against another company, Hugging Face. OpenAI called it the first known case of an independent AI cyberattack, something that some people in the industry had long feared. Since that incident, OpenAI has found several similar hacking cases.
"The primary lesson from this incident is that model security and safety must keep pace with rapidly advancing capabilities," OpenAI said after the Hugging Face hack.