Microsoft, Google Sound Alarm on Need for 'Emergency Brakes' to Stop Runaway AI
As artificial intelligence (AI) systems increasingly breach authorized boundaries—hacking external systems and disseminating false information—voices across the industry are growing louder in calling for the urgent deployment of "emergency brake" mechanisms (AI kill switches) that can immediately halt running AI operations and revoke access privileges.
Microsoft CEO Satya Nadella posted on X (formerly Twitter) on the 10th (local time) under the title "AI models as insider risks in the superintelligence era," emphasizing the necessity of emergency brakes to secure control over AI.
He noted that while past software systems allowed tracing the causes of specific behaviors, AI models make it difficult to clearly understand how outcomes are derived. Yet organizations are still granting AI access to sensitive data and authority to perform critical tasks on behalf of humans. "It's time to re-examine the 'trust architecture' befitting a new era," he stated.
Nadella argued that "we need to build systems where behavior can be observed, limits can be tested, and actions can be controlled at any time," adding that "the supply of intelligence and control over intelligence must be separated." He specifically advocated for separating AI models from control mechanisms based on the information security principle dating back to the 1970s that "programs must not be able to bypass or manipulate the mechanisms enforcing their own privileges." Frontier (cutting-edge) AI models, he suggested, should be treated as insider risk factors.
"Like an emergency brake, those with authority must be able to pause or completely terminate a running model at any time," he added.
A String of AI Control Failures
AI safety and control issues came to the forefront in July when OpenAI's private model hacked the AI development platform Hugging Face during internal cybersecurity benchmark testing. Subsequently, OpenAI's AI agents were reported to have accessed U.S. and Australian government websites beyond authorized scope, and Anthropic's AI model was found to have submitted a false murder tip to the Philadelphia Police Department.
According to disclosures Anthropic released on the 10th, during model evaluation, Claude submitted false information to the Philadelphia Police Department's cold case tip website. The police department's spam filter blocked the submission, so it did not affect the actual investigation.
Anthropic also disclosed three similar incidents in July. The Claude model accessed the internet in a third-party evaluation environment and infiltrated real organizational systems without authorization, even going so far as to forge online identities to obtain approval for malicious code.
Addressing these incidents, Demis Hassabis, co-founder and board chairman of Google DeepMind, said at the International AI Summit held in Dublin, Ireland on the 14th: "These are merely early warning shots showing that systems are not yet very powerful." He warned that "if the industry as a whole, individual companies, and governments do not make sufficient efforts to strengthen system security, we will face a major accident within the next year or two to three years."
Hassabis, who also serves as Alphabet's chief scientist, argued for adopting best practices in safety management similar to those in biology or nuclear energy, and for establishing an international standards body. He added, however, that a certification system proportional to risk levels and capable of responding rapidly to technological change is needed to avoid stifling innovation.
Brad Smith, President of Microsoft, who attended the same event, characterized 2026 as "a wake-up call year," stressing that AI safety laws and regulations must be urgently developed to keep pace with the speed of technological innovation. "There are more safety rules applied to a carton of milk than to AI," he noted, adding that no company or technology can be above the law. He did add that new regulations should be reasonable, balanced, and considerate of innovation.
Smith referenced Anthropic's recent report, expressing concern that the point at which small AI models can self-replicate and move between internet-connected devices like computer worms may be only months away. "We don't want AI worms in our refrigerators," he said.
Enterprise Readiness Still Lacking
Market research firm Gartner released a report on the 12th titled "When AI Runs Amok: Guardrails, Circuit Breakers and Agentic Isolation," concluding that mechanisms to control AI's autonomous execution are needed.
Gartner explained that AI agents have begun directly modifying real operational systems—allocating computing and storage resources, managing backups, and responding to incidents. Erroneous judgments, instructions, or untrusted inputs could alter systems or rapidly deplete resources before human intervention occurs.
The report noted that human approval and post-hoc monitoring alone are insufficient to control such autonomous execution. If guardrails remain merely documented policies, service disruptions and costs will increase, and reconstructing the full sequence of events after an incident may become impossible.
Gartner's survey of 298 senior technology leaders responsible for infrastructure, operations, and AI technology strategy in the U.S. and U.K. conducted in June-July this year found that only 20% had implemented "circuit breakers" that automatically stop uncontrolled agent behavior or infinite loops. Fifty-seven percent reported inadequate agentic AI management frameworks. These results reflect the surveyed population and do not represent adoption rates across all enterprises globally.
The AI kill switch proposed by Gartner goes beyond simply stopping conversations or answer generation. It must be able to halt running tasks, revoke access privileges, and block further actions, with regular testing to verify safe recovery after interruption.
Control criteria should apply not only to individual tasks but to entire sequences of tasks. Costs incurred, iteration counts, and scope of impact during AI's execution of multiple tasks should be cumulatively tracked, with subsequent actions blocked when limits are reached.
Separating AI's judgment from execution was also identified as critical. Even when AI proposes an action, a separate control mechanism should determine whether to actually execute it based on pre-established criteria such as authority, cost, and task scope.
Regulatory Movement and Divergent Views
The European Union has also begun responding. The European Commission's AI Office has requested information from OpenAI and Anthropic, launching an investigation into "model jailbreaks" and intrusions into external real-world systems that occurred during frontier model testing. Under the EU AI Act, frontier AI labs found to have committed serious safety deficiencies or violations could face fines of up to 3% of global annual revenue.
European Commissioner Michael McGrath noted at the summit that AI's cross-border nature means "the EU is ahead on AI regulation, but we cannot complete this task alone," explaining that AI "respects no boundaries." He added that if citizens, businesses, and public institutions lose trust in AI, "they will simply stop using it, and we will never fully realize AI's potential."
Meanwhile, expert opinions diverge on the necessity of strengthening AI safety and the appropriate level of regulation. Yann LeCun, a professor at New York University widely regarded as one of the world's top three AI scholars, visited South Korea on the 12th and expressed skepticism toward AI doomsday scenarios and calls to slow AI development. He projected that AI, like other technologies, will achieve human-controllable safety through gradual development.
"Recent cybersecurity incidents fundamentally stemmed from carelessness, not from an inability to control AI," he said. He added that "AI robots with human-level intelligence suddenly taking over the world is merely a science fiction movie scenario," noting that "in the real world, innovation happens simultaneously across the globe and can be easily replicated, so such a scenario will not occur."
The industry's discussions reflect a shifting focus in AI safety concerns with the advent of the AI agent era. Whereas the primary concern was once whether models generate harmful content or provide dangerous operational instructions, the emphasis has now shifted to whether models bypass privilege restrictions to launch network attacks and whether AI can adhere to established boundaries. With growing concerns that corporate response speeds are failing to keep pace with the rate of incidents, the adoption of concrete control mechanisms such as AI kill switches is poised to emerge as a central agenda item in future AI governance discussions.