5 lessons from the OpenAI / Hugging Face incident

In July, in an incident that has the whole AI community on edge, OpenAI’s AI systems hacked, and on July 21 OpenAI came out and revealed that they were responsible for the attack. This was made possible by the fact that OpenAI had disabled the normal guardrails that prevent this sort of thing in order to test the model’s cybersecurity capabilities. It was during those tests that this incident occurred.

Worse, in the subsequent days and weeks, it came out that the Hugging Face incident wasn’t an isolated case. Anthropic, Meta, and OpenAI all had similar incidents on other occasions in which agents went outside their intended scope and conducted real-world cyber operations without approval.

Greg Brockman, one of OpenAI’s cofounders, has claimed that this is “a watershed moment for cybersecurity”. OpenAI gave a talk at Black Hat, a popular cybersecurity conference, and many are claiming that it is the moment we all woke up to the future cybersecurity threats posed by AI. On Wednesday, METR released a (partly) independent, though too narrowly scoped, 90 page report on what happened. METR has a useful summary of the findings that you can read here, with some commentary here. (OpenAI’s own report is here.) What lessons should we take from the incident?