UN panel calls for stronger safeguards as AI agents advance

The UN-backed Independent International Scientific Panel on AI’s warning followed the hack of the online platform HuggingFace in July by “AI agents” during a test initiated by OpenAI, the company behind ChatGPT. 

AI agents are software that can perform tasks independently and on behalf of a user, compared to chatbots, which are prompted by questions or instructions. 

The panel found that the security breach was the result of a culmination of key risk factors, raising fears that humans will one day no longer be able to steer, constrain or stop AI. 

AI training advancing 

“Researchers have long warned that three conditions could lead to loss of control: a misaligned goal, the capability to pursue it and an environment that allows it. This summer, all three came together in a real system, not a laboratory,” said scientific panel co-chair Yoshua Bengio. 

“Since this is not an isolated observation of misaligned goals, this raises serious questions about the way AI agents are currently trained.” 

The panel’s independent experts stress that the incident provides no assurance that humans can reliably keep AI agents under control, particularly as they become more capable, harder to monitor and better at finding loopholes or hiding their activity.