In the worst-case scenario, that would mean frontier models can not only discover unseen flaws in software, but combine them in an unprecedented, on-the-fly way. While god knows what the hell happens behind closed doors in Donald Trump’s White House, this was reportedly the threat that caused the administration to panic and force two Anthropic models off the market this summer. The attack on Hugging Face certainly appears to have validated that the guardrails being put into LLMs aren’t evolving as fast as their capabilities, at least.
The “allegedly” is because though frontier security models are quite powerful, AI firms also rely on shameless hype to raise countless billions of dollars in investments. Some reviewers have argued they’re more evolutionary than revolutionary. cURL lead developer Daniel Stenberg characterizes LLMs as very good at finding bugs but “not super good at actually assessing the criticality of the problem.”
What can be said definitively is many companies that have gained access to frontier models suddenly start pumping out patches like crazy, like the nearly 1,450 patches Oracle dropped last month.
The saving grace is that AI is at least as effective at defense and possibly even better, according to Brockman. He wrote that frontier models may “shift its [security’s] economics in ways that fundamentally advantage defenders,” like “superhumanly secure code” or generating mathematical proofs that form the foundation of new cryptographic systems and other tools. (Earlier this year, OpenAI did solve an 80-year-old major geometry conjecture, though OpenAI mathematician Sébastien Bubeck told Scientific American the AI’s triumph was more about execution than “something fundamentally new that nobody saw coming.”)
Brockman’s unsurprising 10-step advice to security teams includes, of course, buying more AI. He argues teams should adopt agents and equip them with skills like “static analysis, security-focused code review, vulnerability variant analysis, software supply-chain risk, and other security workflows,” before running security assessments on systems in order of importance.
After that, Brockman wrote, teams should use AI to chip away at vulnerability backlogs, integrate security agents into software development to spot problems as they’re being written, and let agents write “focused” patches directly rather than wait for human review. (This is perhaps capable of causing its own problems—note that security researchers have long warned that agents that go rogue might not be easily shut down.)
To be fair, Brockman did caution to start slowly with automating security operations, which involves triaging incoming security alerts. He suggested starting with read-only scans before escalating to “advisory pull-request scanning, then live alert triage, then automatic closure of narrowly defined false positives.”
