AI agents can’t fix infrastructure failures they can’t diagnose

Presented by Virtana

AI infrastructure failures are hard to diagnose because a fault in one system produces symptoms in several others. When a training job slows down, the GPU dashboard might show contention, the data pipeline report a stall and the storage system start throwing I/O alerts, and each alert suggests a different fix. Infrastructure teams see every signal, and they spend hours working out where the failure started while expensive GPU capacity sits idle.

That search gets harder as enterprises build AI factories that span GPUs, storage, networks, data pipelines, and hybrid environments, because most AI factory monitoring still relies on a separate tool for each domain. And Virtana’s AI Factory Reality Check surveys of U.S. and U.K. enterprise decision makers found that 59% of U.S. enterprises and 53% of U.K. enterprises cannot automatically identify the root cause of an AI workload failure across infrastructure domains.

“Once you get to causality, you can remediate,” says Paul Appleby, president and CEO of Virtana. “If you can’t get to causality, you’re really throwing a dart at a wall to work out what happened and how to fix it.”

Scale makes AI failures harder to trace

Pressure from boards pushes enterprises to show AI progress through deployed infrastructure, so many build first and instrument later, if at all. The legacy tools that handle most of that instrumentation assume stable, bounded systems with predictable service topologies, but in an AI factory, schedulers keep moving workloads across hybrid environments, and resource consumption rises and falls in nonlinear bursts.