AI hallucination nearly triggers US military operation

Military aircraft were already in the air this spring when U.S. officials made an alarming discovery: the intelligence driving an armed operation against a Chinese vessel had been hallucinated by an AI chatbot. The operation was aborted at the last minute, narrowly averting a potential conflict with China, CNN reported on Friday.

The episode underscores a growing concern among military officials and outside experts: As decision-makers lean more heavily on AI, the errors these systems produce can travel up the chain of command before being questioned.

The intelligence report, which circulated during the war with Iran, said the vessel was carrying components for a nuclear weapons program.

The false intelligence originated with a Special Operations Command analyst who queried an AI chatbot to synthesize open-source data with classified signals intelligence. The chatbot misidentified the ship’s cargo manifest. The analyst then used the tool a second time to format the erroneous findings into an official-looking summary, which was circulated across command channels.

The near-miss comes as the U.S military races to integrate AI to accelerate decision-making and maintain its edge over China. The Pentagon has described AI as delivering a significant advantage in speeding up its kill chain so commanders can respond in the right time. But the same speed that makes AI attractive may also allow hallucinations with insufficient human oversight.