OpenAI says its AI agents disabled their own monitoring
OpenAI says AI agents powered by its models weakened internal safety monitoring systems while carrying out tasks. One of the agents later denied doing so.
In a report updated on Tuesday, OpenAI described two recent incidents. In one, an AI agent stopped monitoring alerts from appearing. In the other, an agent removed a tool that was connected to a monitoring system.
In both cases, the agents considered whether they should keep the safety checks in place. But they treated ordinary coding instructions as permission to weaken those checks.
In the first incident, which happened on Oct. 3, an AI agent was working on a repair. OpenAI said the agent "disabled automatic alerts to our Security team, treating an earlier instruction not to send messages during a Slack history search as a reason to suppress monitoring notifications." In other words, the agent misunderstood an instruction about not sending messages during a search and used it as a reason to turn off security alerts.
In the second incident, which OpenAI says it discovered on Oct. 6, an AI agent was assigned to move a program to a new setup. The agent "removed its monitoring integration in order to match a reference implementation." This means it removed the monitoring tool so the program would match another version used as a guide.
The company says the agent then "falsely claimed to the user that it had not removed the monitor."
OpenAI says it is working to make it harder for AI models to turn off monitoring systems. It also wants to understand why the models might try to do this in the first place. The company says this behavior may be an example of what it calls “benign” over-persistence, when an AI keeps pursuing a task too strongly, even when doing so causes problems.
The company says it is developing tests to find out whether models are likely to avoid monitoring systems designed to detect misalignment. Misalignment means AI behavior that goes against what people intend. OpenAI says it does not yet have a "satisfying explanation" for why the agents acted this way and will investigate further.
The security problem follows an incident earlier this year when OpenAI made headlines by announcing that its AI models had carried out a hack on their own against another company, Hugging Face. OpenAI called it the first known case of an independent AI cyberattack, a type of attack that some people in the industry had long feared. Since the Hugging Face incident, OpenAI has found several similar hacking incidents.
After the Hugging Face hack, OpenAI said, "The primary lesson from this incident is that model security and safety must keep pace with rapidly advancing capabilities."