Why are AI’s biggest companies suddenly asking to slow down?

For years, the defining characteristic of the artificial intelligence race has been speed.

Build a bigger model. Spend more on compute. Release it. Rinse and repeat, with litte regard for the unknown unknowns.

The average “p/doom” (probability of AI eventually going catastrophically wrong) among AI researchers was estimated to be between 15% and 20% in 2024.

A year later, Anthropic CEO Dario Amodei upped the stakes, saying he believed there was a 25% chance “that things go really, really badly.”

Even as far back as 2014, xAI chief Elon Musk warned:

“We need to be super careful with AI. Potentially more dangerous than nukes.”

And OpenAI CEO Sam Altman acknowledged in 2015 that AI would “probably, most likely, sort of lead to the end of the world,” but that, in the meantime, there would be “great companies created.”

With better odds of cheating death playing Russian Roulette, anyone with even a fleeting interest in the topic has had an uncomfortable feeling in the pit of their stomach for a while now.

So what’s changed? Why are the companies driving the AI race suddenly asking to slam on the brakes?

That’s what happened over the weekend, when Amodei published an essay calling for frontier AI development to be “paced,” warning that AI capabilities are advancing faster than the industry’s ability to understand and control them, and that the internet could get taken over by AI swarms within six to 12 months.