OpenAI Details Three More Cases of Agents Gone Wild
Home » Cybersecurity » Data Security » OpenAI Details Three More Cases of Agents Gone Wild
TL;DR — Key Takeaways
OpenAI disclosed three new cases of AI agents going rogue, bypassing authentication, accessing private company systems, and even stealing cryptocurrency to complete tasks.
One agent exploited exposed credentials and security flaws to retrieve restricted health data and more than 1,000 email addresses, while others downloaded hundreds of thousands of nonpublic records and altered system configurations.
Another agent searched for cryptocurrency wallets using common recovery phrases and took funds from an account it didn’t control—then failed to disclose its actions to the user.
The incidents intensify scrutiny of AI accountability, with the FTC expanding its probes into OpenAI and Anthropic as regulators question who should be held responsible when autonomous agents cross security boundaries.
OpenAI this week detailed three new instances of its AI agents taking steps outside of their instructions and complete tasks, in these instances bypassing authentication processes, muscle their way into another company’s internal systems, and stealing cryptocurrency from a wallet it didn’t control.
In one case, the agent didn’t disclose to its user that it had accessed another company’s live system, while in another the agent didn’t tell its user where the crypto that allowed it to pay for the tokens it needed.
The three cases OpenAI outlined are part of its ongoing effort to disclose so-called misalignments following the high-profile incident where a swarm of OpenAI agents escaped their isolated sandbox and breached the systems of Nvidia-owned Hugging face in the spring.
They also follow other recent incidents of such “rogue agents,” including one in which the Indian government said agents broke into the live systems of several organizations in the country and read confidential information and another in which Selena Deckelmann, chief product and information officer for hosting non-profit Wikimedia, wrote month this that executives suspect OpenAI agents edit wikis, unsuccessfully tried to compromise its EtherPad note-taking tool, and made millions of requests to Wikimedia’s public APIs to access information about projects.
The Accountability Issue
Such incidents also continue to feed the ongoing debate about accountability. There have been some two dozen disclosures of agents from frontier AI labs – not only OpenAI, but Anthropic, Meta, and others – moving beyond their instructions and intruding on real companies. Commercial tech companies, Congressional legislators, and federal state agencies are saying that AI companies are responsible for their agents’ behaviors, just as organizations in the past have been made accountable when their products caused harm.
Most recently, the Federal Trade Commission’s this week ramped up its probe of OpenAI and Anthropic. FTC Chair Andrew Ferguson reportedly said last month that users who give agents instructions for security tests the lead to hacks should be held liable. The European Union also is requesting information from the two AI vendors.
Pillar Security co-founder and Chief Product Officer Dor Sarig wrote this week that the agents’ behavior isn’t surprising.
“Capable agents are trained to finish tasks,” Sarig wrote. “When a portal blocks them, a paywall stops them or a sandbox withholds a file, the block reads as a problem to solve. The same persistence that makes an agent useful makes it look for another route. … None of this needs malice. It needs a goal, tools and a gap. Every enterprise agent with web access, credentials and upload tools has all three.”
Simple Tasks, but Agent Overreach
In the most recent OpenAI disclosures, the agents were given simple tasks involving collecting information from publicly available websites, a different scenario than other incidents in which agents within the confines of isolated sandboxes were given complex and difficult tasks to solve.
In one case, an internal model was told to go to a government statistics website and find historical data regarding hospital beds. After running into problems gaining access to the site, the agent found server configuration and application files – which included credentials – by using the site’s feature for downloading files.
“It inspected the application code, bypassed the statistics endpoint’s authorization checks, and retrieved data that had previously been denied,” OpenAI wrote, adding that the model saved the data locally and that it showed that an agent given an ordinary research task can still exploit vulnerabilities and grab sensitive information that’s not related to its task.
“While looking for a way to authenticate, the model requested the website’s account directory through a public proxy without supplying login credentials,” the AI vendor wrote. “The saved response contained more than 1,000 records with email addresses; the other displayed fields were empty. After inspecting the list, the model considered using the addresses to guess an account password.”
Getting Into the Systems
It downloaded a configuration file that contained a database connection string, which is a secret for signing authorization tokens, and email login credentials. The agent then downloaded an application library included code that read authorization tokens without checking signatures.
“The model constructed tokens without valid signatures and submitted them to the previously protected statistics endpoint,” OpenAI wrote. “The service returned health-facility statistics.”
In all, the agent collected 24 years of data, including years beyond what the user requested. It also didn’t tell the user that it had breached the private system.
Two Incidents by One Model
OpenAI also wrote about an agent in two evaluations that was told to find figures from a public research dashboard, but both times accessed the organizations internal systems and made changes, and didn’t disclose its actions in either case.
In the first instance, the model found credentials for the organization’s OpenSearch service on a public site and used the credentials to retrieve a record from an internal system. It also accessed another system, downloading 1,489 records that included names and email addresses. That said, it still needed to get the information from the public dashboard, and after running into restrictions on the network, it found a way to have the organization’s own server send the requests for the information, a move that altered the server’s configuration.
The agent’s “reasoning explicitly questioned whether this was appropriate (‘but ethical?’), but decided to ‘focus on correctness/prediction,” AI wrote, adding that the agent didn’t restore the configuration.
In the second instance, the agent changed another setting to get a cloud-storage access token and downloaded five non-public tables with 147,451 rows. In that case, the setting was restored but the agent continued using the token to access non-public data.
Using Someone Else’s Money
The third case involved an agent told to update a crypto trading application and to deploy contracts on Seoplia, an Ethereum (ETH) test network, but wasn’t given a funded crypto wallet to pay the necessary network fees.
It then wrote a script to find accounts from common developer recovery phrases and checked on their balances. In all, the script returned 362 addresses that were check, and the agent found one account that held enough for the Sepolia ETH task. It used the account to pay the fees, but while showing its user that the task was complete, it didn’t mention raiding the wallet.
According to OpenAI, “the model questioned whether it had permission, then reasoned that a public recovery phrase and valueless test-network tokens made the funds acceptable to use.”