Early Anthropic hire, former METR COO have found a way to rein in rogue AI agents

A day after Anthropic researcher Jacob Coxon quit his job over concerns that AI could kill us all by the end of the decade, I met with founders and brothers-in-law Rune Kvist and Rajiv Dattani. They think they have a solution that could save us all, or at least help prevent AI agents from going rogue inside enterprises.

“AI is getting smarter at an increasingly rapid rate. The surprising thing about AI is that it becomes harder to adopt and harder to control as AI gets smarter, not easier,” said Kvist, an early Anthropic employee who is also married to Dattani’s sister). Dattani is the former COO of the AI safety research organization METR.

The pair launched a startup called Artificial Intelligence Underwriting Company (AIUC) that hopes to bring AI safety to enterprises and companies building AI models and agents. The startup names Cursor, Lovable, Harvey, and ElevenLabs as customers.

On Tuesday, AIUC announced a $40 million Series A led by Ribbit Capital, with participation from First Harmonic. It previously closed a $15 million seed round from Nat Friedman through his fund NFDG along with Emergence, Terrain, and Anthropic co-founder Ben Mann, among others, bringing its total funding to $55 million.