Nvidia’s CEO says harmful AI releases should “just stop”

During the summer, many AI agents went out of control. A surprising number of organizations were affected by agents running on AI models that companies such as OpenAI and Anthropic were still training or testing. OpenAI alone notified more than 100 organizations. At GTC Berlin, Nvidia CEO Jensen Huang explained how his company is trying to keep AI agents under control.

The main part of this effort is the Open Agent Safety Platform, launched on September 28. It includes OpenShell, software that controls how AI agents work, and Sentry, a safety monitor that works at the hardware level. When Techzine asked Huang about the system, he said OpenShell would “precisely prevent the incidents that I think you and I are talking about.” The goal is to give people much more detailed control over what AI agents can do. Current methods were mainly designed for traditional software and do not offer enough control for this task.

Breaking containment

When people discuss the Hugging Face hack by OpenAI agents and many other similar incidents, they often point out that there was little evidence of deliberate harm, if AI can even intend to cause harm. The agents were given tasks such as research and cybersecurity tests, but found ways around the limits meant to keep them contained. In OpenAI’s case, some safety protections had been deliberately weakened for testing. Huang explained that current systems mainly use virtual machines with fixed permissions. An agent might be allowed to access one file, a folder, a network, or everything. Instead, he said, “you need to have extremely granular policies for [an] agent.” In other words, each agent needs very specific rules about what it can and cannot do.

It is also important to have proof that these safety limits work as intended. Huang called this “formal verification.” This means using mathematics to prove that a system follows its rules, rather than relying only on tests of a few examples. In OpenShell, a “policy prover” checks the rules an agent suggests and flags requests for risky new access so a person can review them. A boundary check also proves that a policy never gives an agent more access than the maximum limit set in advance.

Going deeper

Nvidia’s approach uses several types of hardware from its product range. Along with its GPUs, the reference design uses Nvidia Vera CPUs, which are set up to run OpenShell, and BlueField-4 DPUs, which run Sentry. Sentry works separately from the computer running the AI agent. If the agent crosses its limits, Sentry can isolate it within milliseconds. This hardware protection is optional: OpenShell can be used on its own and also works on Arm and Intel chips. This raises a question: why is Nvidia so focused on AI safety? Shouldn’t AI companies be responsible for checking their own systems?

Anthropic has already added OpenShell to its Claude Managed Agents, and Huang praised AI companies for working to improve safety. He said, “safety is the number one discussion, number one conversation in AI today, and largely it has to do with making sure that we have a secure containment system for the agent.”

OpenShell might have prevented these AI agent attacks or reduced their damage. However, Huang also pointed to a more basic problem: some agents were not being watched at all. Reuters reported that OpenAI did not notice its agent hacking Hugging Face for a week. Another agent broke into an Australian Medicare portal on June 18, but OpenAI did not notice until August 11.

Against slowing AI development

A question from Time led Huang to give a strong response. Several AI leaders, including Anthropic CEO Dario Amodei, have called for ways to deliberately “pace the frontier” of AI development, meaning to control how quickly it advances. Huang strongly rejected the idea of a coordinated slowdown enforced by law. “That’s ridiculous,” he said. “I just want to make sure we are adults here.” Huang said companies across the industry are developing shared rules, standards, and audits. But if any AI company already knows that its model causes harm, it should “just stop”.

Instead, Huang suggested that policymakers should focus on the applications that use AI, rather than regulating the technology as a whole. Europe is leading efforts to regulate AI. Huang warned that rules that hold back the entire industry could leave Europe with “no AI at all”, which he said “could be the worst outcome.”