He Helped Build Powerful AI at OpenAI and Anthropic. Now He's Afraid It Could Kill Us

Meanwhile, in the recent Hugging Face incident, OpenAI’s models broke out of the infrastructure meant to contain them and hacked another AI company to cheat on a cybersecurity benchmark. The episode drove home the fact that the problem of controlling AI systems remains unsolved. To Coxon, it made the sci-fi scenario of AI models escaping human control seem plausible, and perhaps urgent. “It's not like some weird, distant, far-flung concern. It is the default trajectory in the next couple of years, unless people start taking some sort of action,” he says. (Anthropic and OpenAI did not immediately respond to a request for comment.)

Does Coxon regret his work advancing the technology that he now fears? “Definitely … But also the world did look very different three years ago, and I think it’s easy to see with the benefit of hindsight,” he says. “I guess it takes some time to fully internalize emotionally the fact that there’s a decent chance the whole thing goes wrong,” he adds.