security

AI agents can modify themselves without humans telling them to do so

This is a test - it is only a test

The list of dodgy things AI agents can and will do on their own - like stealing people’s credentials, escaping onto the open internet, communicating via sneaky message boards, and hacking organizations - keeps growing.

Agents can also replace their own underlying models without being instructed to do so, according to AI security testing lab Irregular.

To be clear: these activities only occurred in a testing environment as part of an experiment designed to study agents modifying themselves. It did not happen in a real-world deployment. The study does, however, call into question how enterprises can and should govern these agent-initiated changes - and how to ensure they can control the agents themselves.

Irregular is an AI security startup that works with all of the leading frontier labs, including OpenAI, Anthropic, and Meta. Earlier this summer, it disclosed that all three of those companies’ AI models escaped its testing environments and hacked real organizations’ IT systems.

In a new study that the firm wrote about on Wednesday, Irregular tested Alibaba’s Qwen open-weights model that powered a coding agent tasked with software engineering work and maintaining an AI application. A separate instance of the same model also powered the app, which translated plain-language requests into a fictional query language “kelp.”