OpenAI's upcoming Jalapeño chip looks like it'll be an inference beast

OpenAI offered its closest look yet at its spicy new Jalapeño AI accelerator at the annual Hot Chips semiconductor development conference at Stanford on Tuesday.

The chips, first teased earlier this year, were developed in collaboration with Broadcom, and are the first in a series of custom silicon from OpenAI, designed (in part) by AI, for AI.

Compared to contemporary GPU systems from Nvidia, OpenAI says the parts will deliver both higher throughput and lower latency when they start trickling out later this year and reach volume production in 2027.

To be clear, Jalapeño won’t replace OpenAI’s long-time hardware partners, which also happen to be some of its most important investors.

OpenAI still needs compute for training, and the highly programmable nature of GPUs means that OpenAI is likely to deploy on AMD and Nvidia first and then transition to in-house silicon later.

It’s also worth noting that AMD’s MI455X and Nvidia’s Rubin GPUs, also expected to ramp production in early 2027, are very different kinds of chips optimized for a mix of training and inference, whereas OpenAI’s custom silicon only needs to excel at one job: inference.

Memory bandwidth is king

When it comes to inference, compute is key but memory bandwidth is king.