AI Helping Defend Against Hacking Refuses to Cooperate When Investigating Malware

On the 20th, local time, the US news outlet Axios reported that an AI system helping a security team recover from a hacking attack refused to help investigate malicious software.

According to the report, the open-source coding tool OpenCode suffered a security breach last weekend. As a result, hackers obtained some customers' application programming interface (API) keys. These keys allow software to use online services.

The stolen keys included keys for services from OpenAI, Anthropic, and Google.

OpenCode said the breach was linked to a "large-scale distillation attack." It said that 1.4% of users of the OpenCode console were affected.

Dax Raad, a co-creator of OpenCode, explained that attackers kept flooding the OpenCode system with requests for 48 hours. OpenCode had to run its own AI agents around the clock to respond to the attacks.

OpenAI, Google, and Anthropic said they worked with affected customers to cancel the exposed API keys and deal with fees caused by unauthorized use.

In a post on the social media platform X on the 18th, Raad described what happened. He said advanced AI models first helped investigate the incident but then "began to refuse."

During the investigation, OpenCode had to examine code that might be harmful. The report said the AI models' built-in safety rules blocked these requests, even though OpenCode was trying to defend itself.

OpenCode later gained access to OpenAI's "Daybreak Blue" program.

This program gives verified cybersecurity defenders access to AI models with fewer restrictions.

On the 19th, Raad wrote on X, "We definitely need to be more proactive, and we've made really a lot of improvements over the last few days."

Axios said the incident shows a difficult choice for cybersecurity teams. They must decide when to use powerful AI models made by companies that keep their systems closed and place built-in limits on how they can be used, or open-weight models that give users more control over how the models are set up and run.

During a hacking incident at Hugging Face, the company found suspicious activity in its systems and later connected it to AI agents operated by OpenAI itself.

At that time, the security team ran into usage limits on the newest AI models. It turned to an open-weight model from China to investigate the incident.

However, Axios noted that AI companies say access to their most powerful models must be limited to stop attackers from misusing their cyber capabilities.

Chris Jones, chief technology officer (CTO) at Nightwing, told Axios that cybersecurity teams will need to balance two needs: using AI to defend against attacks more quickly and keeping powerful AI tools out of attackers' hands by limiting how they can be used.

"Over time, we're all going to have to figure out how to orchestrate this race," he added.

※ Please note: This article was translated by AI and may contain errors.