During the Hot Chips conference on Tuesday, OpenAI presented an in-depth overview of Jalapeño, unveiling the initial benchmark results for its new system. When evaluated using SemiAnalysis’ InferenceX benchmark, Jalapeño demonstrated superior performance, achieving higher tokens per user and greater throughput per kilowatt compared to existing cutting-edge inference processors.
Richard Ho, OpenAI’s head of hardware, stated in a press call that “the results clearly indicate a substantial performance improvement over current state-of-the-art technology.” He elaborated that “Jalapeño is capable of handling more AI tasks per unit of power and delivers quicker responses. It offers high efficiency for numerous users while also maintaining very low latency.”
It’s worth noting that this comparison was made against an Nvidia Blackwell system; however, competing technologies might have evolved considerably by the time Jalapeño is fully deployed. Ho projected that Jalapeño’s deployment would commence in late 2026 “in very limited quantities,” with broader deployment expected in 2027.
Jalapeño, initially revealed last October, was developed by OpenAI through a close partnership with Broadcom, with OpenAI’s proprietary models aiding the development. The company intends to evolve Jalapeño into a multi-generational platform, enabling the synchronized development of AI products, models, chips, and memory.
This comprehensive, full-stack methodology allowed OpenAI to target particular stages in the inference process that frequently create inefficiencies. Specifically, Jalapeño is engineered to reduce delays during the prefill and communication stages of processing, which OpenAI identifies as common bottlenecks.
In a blog post detailing the results, the company stated, “Our design for Jalapeño focused on minimizing data movement and communication delays.” They added, “This ensures that the model state, including the KV cache utilized during response generation, can be explicitly localized and maintained while the system optimally engages its compute, memory, and networking resources for each phase of inference.”