OpenAI and Broadcom built a custom chip for running OpenAI's models — not training them — in about nine months. It's codenamed Jalapeño, early lab numbers reportedly show a meaningful jump in performance per watt over current alternatives, and it's slated to go live in production late this year. None of that is the interesting part. The interesting part is that the two biggest AI companies in the world just spent nine months and real engineering effort on a chip whose entire job is running queries more cheaply, not more of them.
Training was never the expensive part, long term
A model gets trained once, or retrained periodically. It gets queried billions of times a day. As agentic and generative AI workloads scale, inference — not training — dominates the actual cost of running an AI product. That's the whole economic case for a purpose-built inference chip: a general-purpose GPU is flexible, and flexibility has a cost when you're running the same kind of workload, at the same kind of scale, for years.
Performance-per-watt isn't an abstract engineering metric here. It maps directly to two line items every AI infrastructure budget already has: power and cooling. As products go from millions to billions of daily interactions, small efficiency gains compound into real money — which is why custom silicon, once something only Amazon, Google, and Microsoft bothered with, is now a conversation worth having for any company running AI at meaningful scale.
The chip doesn't matter to you — the trend does
You're not going to design your own inference chip, and neither am I. What matters is what this signals about where the hardware market is going, and it changes a few things worth asking about the next time you're evaluating a cloud AI vendor:
- Pricing pressure. As hyperscalers deploy more efficient inference silicon internally, expect that efficiency to show up eventually as downward pressure on inference API pricing — it usually takes a while to pass through, but it does pass through.
- A different vendor question. Ask about a vendor's inference efficiency roadmap specifically, separate from their training compute story. Those are two different conversations, and most sales conversations only cover the second one.
- Workload placement. If you're running AI on-prem or hybrid, your highest-volume, most predictable workloads are the ones worth reassessing for specialized inference hardware. Experimental or low-volume workloads aren't — the setup cost isn't worth it yet.
Don't lose the interoperability thread
Custom silicon built on closed, proprietary interconnects risks splitting the AI hardware ecosystem into islands that don't talk to each other. The counterweight to that is the parallel push toward open standards for AI infrastructure, which is what lets an organization mix a custom inference chip, a general-purpose GPU, and a third party's networking hardware inside the same data center without getting locked to one vendor's full stack. If you're negotiating a multi-year infrastructure contract right now, interoperability support is worth weighting as a real criterion, not a nice-to-have.
What to actually do with this
Model your total cost of inference, not just the licensing line — power, cooling, and networking overhead belong in that number, because that's where the Jalapeño-style gains actually show up. And the next time an AI infrastructure vendor pitches you on raw compute, ask them the efficiency question directly. Their answer, or lack of one, tells you more about where they're headed than their benchmark slide does.