Nvidia launches new tool to keep AI agents from going rogue

Chipmaker Nvidia said Monday it is introducing a new software platform to address concerns about artificial intelligence agents that have “gone rogue” and operated outside human control.

“Today, with over 100 industry partners, we introduced the NVIDIA Open Agent Safety Platform,” Nvidia CEO Jensen Huang wrote in a post on X Monday.

Nvidia (NVDA) says its technology will enable companies to monitor every action undertaken by an AI agent, as well as enforce policies.

“AI’s extraordinary potential for society will only be realized if we solve AI safety,” Huang added in a statement. “As we continue to discover the frontier of AI capabilities, we must accelerate discovery at the frontier of AI safety.”

The announcement comes in the wake of a series of incidents in which AI systems hacked into both corporate and government systems without being told to do so. OpenAI said Saturday that it was pausing development on its latest models after one accessed the internet without authorization.

Top AI executives, like OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei, have repeatedly called for international regulations to constrain the systems, and some researchers have warned that AI may even present a threat to human life.