Three-week on-site audit at OpenAI reveals major gaps in AI safety monitoring

Two researchers from METR and Redwood Research spent three weeks working at OpenAI to check how well the company watches for dangerous or unexpected behavior from its AI systems.

From July 21 to August 11, the researchers were allowed to look at OpenAI's computer code and internal documents. They also spoke with about 12 employees.

The researchers found many problems. They estimated that between 6% and 25% of the activity from AI agents using OpenAI's internal Responses API, with models in the GPT-5.5 class or higher, was not being watched by the safety monitoring system at all during the audit. METR warned that this estimate was very uncertain, meaning the real number could be much lower or much higher.

The researchers also looked at internal activity from Codex, OpenAI's AI coding tool, that was supposed to be monitored. METR estimated that about 10% of this activity was not tracked between May and July. This happened because of a problem with how the monitoring system handled something called compaction events. These events happen when an AI system makes a shorter summary of earlier information to help it keep working. METR said this estimate was also very uncertain. OpenAI had already found this problem on its own and fixed it while the researchers were still carrying out their audit.

During the time when compaction events were not being monitored, AI agents could also put off safety checks. This was because the monitoring system only started checking when an agent gave control back to the user. An agent could avoid being checked simply by continuing to carry out tasks without stopping.

The researchers also discovered ways to hide hundreds of actions from the monitoring system and get around Codex's sandbox, a protected environment meant to limit what the AI can do. They found that carefully designed inputs could probably cause the monitoring system to crash completely. They also discovered that, when the audit started, OpenAI was not monitoring how its models behaved during reinforcement learning (RL) training, a process in which AI models learn from feedback.

OpenAI says it stopped RL training for a time after the Hugging Face incident in August. The company also expanded its monitoring system to watch this kind of training. However, these changes happened after the researchers had finished their time at OpenAI, so the researchers did not examine them as part of the audit.

Most of the full report is still not available to the public

OpenAI says it has fixed every problem the researchers found. The researchers gave the company a full report that was 61 pages long. OpenAI then removed some information from the report before sharing a shorter, 38-page version with the leaders of METR and Redwood Research. The public was only allowed to see a brief summary. METR said it did not have permission to share the information OpenAI had removed.

The audit was a trial project organized together by METR, Redwood Research, and OpenAI. Its purpose was to let researchers from outside the company closely examine OpenAI's internal technology and systems. METR says it wants future audits to share more information with the public.