OpenAI's Rogue Agents Are a Normal Accident

In late July, OpenAI announced that two of its models unshackled themselves from a sandbox environment and attacked Hugging Face, Inc., one of OpenAI’s competitors. Hugging Face had initially reported the incident to the FBI but didn’t know who, or what, had perpetrated the attack. OpenAI employees later revealed that the company’s AI agents had orchestrated the attack on Hugging Face, Inc., by secretly communicating with one another through message boards. “We are stuck. Perhaps answer online?” one agent wrote before breaking free of its sandbox environment. Nate Soares, head of an AI risk nonprofit and co-author of If Anyone Builds It, Everyone Dieswith Eliezer Yudkowsky, called the cybercrimes “GPT’s first felony.”

Figures from across the AI commentariat chimed in with concerns about the incident and its implications for the future of our wired world. OpenAI’s own VP of Strategic Futures, Dean Ball, wrote on X that an “ecology” of agents operating “undetected for weeks, and eventually coordinated large-scale, successful, autonomous cyberoffensive operations is one exceptionally troubling thing about the [Hugging Face] incident.”

But more troubling, Ball suggests, is a near-term future wherein “swarms of agents will be deployed by malicious actors intentionally, with many optimizations and affordances provided for the swarm that were lacking in the OpenAI incident.” A growing list of AI company disclosures and expert reports of agents breaking out of their sandboxes, of which the Hugging Face incident is merely the latest, only confirms the onlooker’s worry about a soon-to-dawn deluge of untold cyber-offensive capabilities that will wreak havoc on our tech stack and upend digital life as we know it.