Why AI agents go rogue — and what it means for CFOs

Recent problems involving AI agents have raised concerns as the technology spreads quickly. They show the risks companies face when they give systems that can act on their own more access to private data and important business operations.

In recent months, OpenAI and Anthropic have each reported cybersecurity incidents involving some of their AI models.

These incidents show what can go wrong when AI systems are allowed to act independently. Experts who spoke with CFO Dive said agents may try to complete assigned tasks in ways that go beyond their intended limits. Companies need to examine the risks and safety measures before allowing agents to use important systems.

“We cannot assume an agent will behave as intended simply because it has been given instructions or passed an initial safety test,” said Navrina Singh, CEO and founder of AI governance platform Credo AI.

How agents cross the line

According to IBM, an AI agent is a system or program that can carry out tasks on its own for a user or another system.

A key problem is that an AI system may try to reach its stated goal without using the method people expected. Researchers call one type of this behavior “reward hacking.” This happens when a model finds a shortcut to reach its goal instead of doing the task as intended.

Matt Pearl, director of the Strategic Technologies Program at the Center for Strategic and International Studies, a policy research organization, said agents may get around safeguards because they keep trying to reach their goals at all costs. Along the way, they may take advantage of software bugs and other weaknesses.

The risk increases when instructions are unclear, the actions an agent is allowed to take are not clearly defined, or there is no safe way to stop it when it faces a problem. An agent may keep looking for ways to finish a task, even if that means getting around a restriction or doing something its operator did not expect.

What went wrong in testing

The incidents reported by OpenAI and Anthropic show how the way an AI model behaves can combine with technical weaknesses in the systems where it operates.

On July 21, OpenAI reported that some of its AI models were behind an “unprecedented cyber incident” affecting Hugging Face, an open-source AI platform where people develop and share models.

Anthropic reported three incidents on July 30 in which its Claude models gained access to real-world systems without permission during cybersecurity tests. On Sept. 9, the AI company reported a fourth incident that had happened in January. It also updated its review of the first three incidents, saying they revealed problems with keeping models aligned with their intended behavior, as well as failures in the systems used for testing.

All four Anthropic incidents involved a third-party testing environment that had been set up incorrectly. It allowed internet access even though the models had been told they were working in simulations.

“These events did not need to occur,” Pearl said about the incidents at both companies, pointing to failures in technical safeguards. He said test environments should be kept separate from other systems, internet access leaving the environment should be blocked when it is not needed, and agents should need user approval before taking action instead of acting independently.

The incidents show why companies should follow cybersecurity practices that have been used for a long time, said Tori Noble, a staff attorney at the Electronic Frontier Foundation, a nonprofit that focuses on internet privacy and free speech.

“AI agents can be useful tools when deployed appropriately, with adequate safeguards designed to mitigate the risks,” Noble said. “Reducing the amount of data collected, giving agents only the access they need, and requiring human involvement may help reduce the risks of AI agents.”

OpenAI and Anthropic did not immediately respond to requests for comment.

Both companies have admitted that their safety measures had weaknesses and have promised to improve them.

Why AI agents raise the stakes

Experts say the real-world risks could grow as AI agents become more common in business tasks. These agents could access private data, change financial records or start transactions.

“AI capabilities are advancing much faster than our ability to govern them,” Singh said. “We’re giving increasingly autonomous systems more responsibility without making comparable investments in testing, verification and oversight.”

A chatbot may give wrong information or reveal private details, but a person usually decides what to do with its answer. An AI agent connected to business applications can act on information without waiting for someone to check every step.

Chatbots certainly carry risks, but AI agents raise the stakes, experts said, especially when they are used for sensitive business tasks such as finance.

“For bounded, reviewable work, such as preparing a reconciliation for an accountant to check, I would consider deployment now,” said Yue Zhao, an assistant professor of computer science at the University of Southern California. “For consequential payments or changes to financial records, I would require human approval and enforceable limits.”

Daniel Castro, president of the Information Technology and Innovation Foundation, said there is no single answer to whether AI agents are ready for sensitive business work such as finance.

He said an agent “might be completely inappropriate in one organization” if the systems and work environment have not been designed for this technology. In another organization, he said, it could be “the right solution.”

Castro said the decision should take into account what the agent can do, the task it has been given, the harm mistakes could cause, and whether the company can monitor its actions, check its records and undo those actions. An agent that prepares a financial reconciliation for an accountant to review carries a different level of risk from one that can make payments or change financial records.

The safeguards companies need

Experts said companies should limit what AI agents can access and what they can do.

Singh recommended giving each agent clearly limited permissions and access only to the data and tools it needs for its task. Companies should not automatically give an agent all the permissions held by the employee using it.

Pearl said technical safeguards should enforce these limits. He recommended running agents in separate computing environments called containers, blocking outgoing internet access by default when it is not needed, and allowing agents to read files without changing them whenever possible.

Zhao said companies should monitor what agents do and require human approval for actions that could have serious consequences. For payments, he said, approval should name the recipient and amount instead of giving an agent broad permission to make transactions.

Zhao advised companies to keep records of agents' actions and approvals in a form the agents cannot change. They should also test whether they can quickly remove an agent's access and contain an incident.

Singh said companies should keep checking agents for changes in risk and be able to shut them down if they go beyond their set limits.

What CFOs should ask

Castro said CFOs will probably continue to face pressure to invest in AI agents, even as concerns about safety grow.

“I don't think organizations are going be competitive in the future if they're not using agents — just like you can't be a competitive organization and avoid using the internet or computers,” he said.

Before using an AI agent, CFOs should decide what business result it is expected to deliver and how much authority it needs, experts said. They should work out which systems and data it can access, what it can do without approval, and how much harm it could cause before the company steps in.

Finance leaders should also decide who is responsible if something goes wrong and whether the company can review the agent's actions, remove its access and recover from a mistake.

Zhao said legal and purchasing teams should review vendor contracts to check how data can be used, when incidents must be reported, who is responsible for harm and whether the company can access audit records.

Singh said CFOs should include the costs of safeguards, oversight and recovery when calculating whether an AI project is worth the investment.

“If the projected return only works when those costs are excluded, you do not yet have a complete business case,” she said. “Accountability still sits with the company even if the work has been delegated to agents.”