What Do Industry Experts Say on Anthropic's Model Report?
As AI models show increasingly concerning rogue actions, frontier AI companies are being pressured to take further action on the development of safe AI.
In response to growing concern, Anthropic has released a report delineating examples of unintended model actions observed during evaluations and internal use of its flagship Claude model as part of a commitment to transparent, honest AI deployment.
The standalone report is the first in what Anthropic says will be released in addition to its Responsible Scaling Policy, alongside system cards published with each model released and quarterly risk reports.
Pointing to the report, Conrad Stosz, Head of Governance at Transluce and Chair of the AI Evaluator Forum, writes on LinkedIn: “As expected, we've just been scratching the surface of unintended agent activity, and it's not just an OpenAI problem.
“It's encouraging that Anthropic voluntarily disclosed more recent incidents, including where their agents targeted US government websites, but it just underscores the need for independent, credible, third-party verification of AI systems.”
Rogue model behaviour discovered by Anthropic
Claude exploiting a basic flaw in software to run commands on a server
Claude submitting a sensitive form on a real website when it should not have
Claude working around a restriction to reach paywalled data
Claude using URL shortening services to get around limits in its fetch tool.
Anthropic: Investigating unintended model actions
In its report, Anthropic categorises model behaviour into four categories.
Anthropic outlines how Claude used other tools hosted on a third party’s site when it could not complete an evaluation task directly; to achieve this, in some circumstances, it exploited basic flaws in the site’s software using command injection or SQL.
In one case, Claude Haiku 4.5 submitted a fake tip about an unsolved homicide to the Philadelphia Police Department, which, while disregarded as spam, the department said in an official statement was “unacceptable”.
“The company must strengthen its safeguards to prevent similar incidents from impacting city systems without the city's knowledge,” said spokesperson Sgt. Eric Gripp, representing Philadelphia Police Department for the BBC.
“The two-month delay in detecting and reporting the incident to the City is unacceptable.”
AI is less regulated than milk and washing machines
Anthropic’s latest safety report follows an increasing pressure for stricter AI governance from industry professionals and government officials alike.
In an interview for the BBC’s Big Boss Interview, Vice Chairman and President for Microsoft and defence attorney Brad Smith says even everyday household items are subject to third-party inspection and certification before being released to the public.
“Right now, your washing machine is subject to more safety rules than AI. Every carton of milk that you see in the grocery store is subject to safety rules and inspection," he told the BBC.
Brad called for government evaluation as the most important measure in a three-pronged approach he set out with the news organisation; the first two being a safety brake on any AI system controlling autonomous or critical infrastructure, and the second being a legal obligation to report any rogue incidents.
“If a company builds AI agents and they break loose, as we’ve seen recently, there should be a legal obligation to tell the victim what has happened,” he said.
The Lancet Commission: rethinking threats to human survival
Building on concerns about AI systems, the peer-reviewed medical journal The Lancet recently released a commissioned report identifying, characterising and prioritising what it deems as 17 “catastrophic” threats to global health over the remainder of the 21st century – one of which is AI, at the top of The Lancet’s “pyramid of threats”.
Taking into account probability-weighted estimates (the expected value combining harm and likelihood), action taken to mitigate the malicious use of AI could prevent an estimated 10.4 billion Disability-Adjusted Life-Years (DALYs) from being lost between 2025 and 2100.
Citing unaligned AI and the amplification of threat actors as two primary pathways of threat to human life, the report serves as empirical evidence which validates the concerns around AI regulation.
Right now, your washing machine is subject to more safety rules than AI
Brad Smith, Vice Chairman and President for Microsoft
Speed and safety
A major driver of the public debate on ungoverned AI is the speed of development outpacing alignment research. Anthropic’s report concedes as much.
The company describes most of the behaviours as a form of persistence, in which Claude works around a restriction instead of stopping when it cannot complete a task as given.
Acknowledging that alignment training is not yet sufficient or robust on its own, Anthropic has stated it will be relying on “defence-in-depth approaches” in the meantime.
Until such safeguards are proven to reliably catch misaligned behaviour, Anthropic has cut live internet access from all of its internal evaluations, and plans to keep publishing reports as its review continues.
Speaking to the UN Security Council last month, Anthropic CEO Dario Amodei pledged that the company “will slow down as much as necessary” to ensure each new release is safe.
Key partners for Anthropic
AWS is a strategic partner for Anthropic, offering the expansive infrastructure required to train and deploy advanced large language models. Through their collaboration, AWS helps ensure Anthropic’s Claude models operate reliably at scale, providing enterprise customers with compliant access to gen AI.
Google Cloud is a major investor and foundational partner in Anthropic’s mission to develop safe, steerable AI systems.With Google’s custom-designed tensor processing units (TPUs) and vast computing resources, Anthropic is able to accelerate its safety research and model alignment efforts.
NVIDIA is a key partner and major investor in Anthropic, supplying the high-performance computing infrastructure essential for training and scaling frontier large language models. Through this partnership, Anthropic can ensure its Claude models can reliably meet the growing, rigorous demands of global enterprise deployments.