We must recall open-ended AI agents with internet access from the market, now

Breaking new from The New York Times, the latest of many agent-caused incidents that are happening with frightening regularity, but this time from Anthropic and fairly serious:

I have long felt that OpenAI is handling this inadequately. Seeing the same kind of incidents at Anthropic makes it absolutely clear that the current generation of synthetic agents simply cannot be trusted. Until they can be fixed, they should be removed from the market, just like a car with defective brakes.

§

Ezra Klein’s new interview with recently departed AI employee David Robinson only furthers my sense that these companies are in wildly over their heads:

Quoting in part:

And I don’t think that we or our peers — really, anyone in the industry — are being safe enough. I think OpenAI and its peers are now producing a technology that is more capable and poses more risk than what was being made even six months ago.

I’m not a scientist. I’m a writer. What I know is what the execution environment looks like for our safety work, and we’re operating — and I believe the industry is operating — like a start-up still, more so than makes sense. Not maybe completely like a brand-new start-up, but we’re too close to that end of the spectrum for really dangerous systems that could pose risks — loss of control is one example. If that did happen, we’re talking about a harm that’s much larger, for example, than a single nuclear power station melting down. And the internal controls and safety and redundancies are just nowhere near what the world expects for a nuclear power facility.

Now, some of this is known, right? OpenAI has publicly reported on safety problems. Obviously, Hugging Face, but also other ones, including more recently. And Anthropic, by the way, also has reported, including an instance in which their safeguards were accidentally misconfigured.

So I think people do have some evidence already externally that things are not as they ought to be. But I also think if you were watching from the outside, you might imagine that we have a more robust safety setup than we actually do.