OpenAI 硬件负责人 Richard Ho 谈 Jalapeño AI 芯片的设计与流片

Jalapeño. Credit: OpenAI A new AI chip is making waves in the industry and it’s likely to keep doing so for years to come. In late August, OpenAI presented its Jalapeño AI inference chip at the Hot Chips conference to widespread acclaim. SemiAnalysis InferenceX benchmarks showed the design outperforming its rivals in performance per watt, including Nvidia’s Blackwell, on a number of open-source models such as GPT-OSS, DeepSeek R1, and Kimi K2.5. The rankings could change once the newest Vera Rubin AI systems enter the picture. But building such a high-performing first chip so quickly is a big achievement. I believe OpenAI’s Jalapeño is the most promising chip program in the latest wave of non-GPU AI chips. Key Context spoke with Richard Ho, VP of Hardware at OpenAI, who led the Jalapeño chip team. We discussed how he joined the frontier AI model startup, how Jalapeño was built, what makes it special, and the future of the chip program. As always, I enjoyed talking with one of the top engineers in the AI and chip industry. This Q&A is packed with technical details and insightful nuggets on where chip design and memory technology are going. I’ve bolded the remarks I found most important. Here are lightly edited highlights from our conversation. Tae: Could you share your background and tell us how you were recruited to OpenAI? Richard: My background is in computer architecture. I’ve been working on supercomputers for many years, including at D. E. Shaw Research, where we were building molecular dynamics supercomputers. I was then recruited to Google, where I worked on the TPU from the first generation. I was the first engineer working on it. I’ve always loved AI/ML. I actually started my PhD in artificial intelligence. I came to Stanford to work on artificial intelligence. At that time, it was expert systems, not neural nets. It was not a good place to be. It was the start of one of the AI winters and funding for AI work had dried up. So I moved into hardware. I’m very happy to be able to get back [to AI]. The way I got back into it was through Google. I had been poking around to see where the next generation of technology was. I’d moved over to a photonics company, Lightmatter. It was doing photonic communication as well as photonic compute. I think that really exposed me to where AI and ML inference was going to head in terms of technology, and where the advances needed to be to get perf per watt, lower cost, and everything else. At that point, I think Sam [Altman] and his people reached out to me to have a conversation about where I thought things were and where they were going. At the time, OpenAI was not really into hardware, but Sam has always been interested in hardware. He’s invested in a lot of small startups through Y Combinator and other places. So he’s always known that infrastructure is super critical to delivering intelligence. We hit it off in the conversation, and then I basically got recruited in. At that point, there was no one else here. We didn’t know what we were doing. Sam did not say “go build a chip” straight off. He said, let’s evaluate. And I said, let’s evaluate, and let’s see what we could do that would benefit that infrastructure. And that’s what we’ve done. After doing some careful analysis of our workloads, our models, the software stack, where hardware was, and where it was trending, we decided it was worth doing it ourselves to deliver full-stack inference capability with lower cost and lower power for our users. I think that’s the ultimate goal and the mission of OpenAI. To deliver intelligence for the benefit of all humanity. That really fit with my thinking about what I wanted to do and where I wanted to be, so this seemed like a natural fit for me. Tae: The benchmark results for Jalapeno look phenomenal. For a general reader, at a high level, what sets Jalapeno apart from a TPU or GPU? How did you approach the design? Starting with a blank slate is a really key part of it. In the software world, for example, you get revs and rewrites of code because people learn. They do the first rev, they learn from a lot of mistakes, and they say, oh, we could do this better. They start from a clean slate, almost, so rev two is a rewrite. Equally, on hardware, we were the first real, serious effort thinking from first principles about what it would take to serve LLMs and transformers this way at scale. So we were able to look at it with no legacy, from the architecture viewpoint, and see what could be done to improve energy efficiency. Energy efficiency boils down to data movement and the reduction of data movement. We’ve focused heavily on that. We asked ourselves, what is the minimum necessary for data movement? There’s been a lot of research in computer architecture at universities about putting compute with memory. They call it PIM, or processing-in-memory, or CIM, compute-in-memory, and that’s still advancing. But we took an intermediate step where we hooked up our HBM with our cores in a very technical way, and that gave us a lot of energy efficiency and a lot of speed as well, in terms of token latency. The downside is that the programming model becomes more difficult because you have to account for the management of the memory, and that’s always been a problem. Other architectures have had that problem. But we were also betting that AI models were going to get good at this, and indeed they did. That allowed us to have a reasonably good programming model while taking advantage of this rather difficult way of mapping data to memory that gets you very efficient processing. A lot of things came together at the right time for this to happen. I think that’s the secret sauce. Sometimes you have to be lucky with timing. Sometimes things are too early, sometimes too late. In this case, things came together at just the right time. We had a team available. We had a motivation. We had a need. All these things came together to get the Jalapeno design up and running. Tae: I was at Hot Chips when you presented the HBM slice architecture, where each HBM bank is linked to a core. Can you expand on that? Why does it benefit AI? The key differentiator between that architecture and a standard GPU or even CPU architecture is that the memory space is now split up. Each dedicated core, and we have 64 of them, can directly access and address its own bank of memory in the HBM. To address memory in other places, it has to go through a network, which is a lot slower and involves more indirection. In a general CPU or GPU, the memory space is shared, which means there’s a single address space. In Nvidia speak, a core is an SM. You can access any memory from anywhere in the chip, which makes programming easy, but there’s a huge hardware overhead needed to manage that shared memory, manage access to it, make it coherent, and so on. We got rid of all that, and by doing so we were more efficient and faster. But again, you then have to manage the memory map. Your core can mostly access only its own memory bank, and to go anywhere else you have to do some extra work. So the trade-off was, does that pay off or is it too expensive? When we did the analysis with the models, we found that it paid off big time. We were then able to lean into that architecture and say, let’s make it work, let’s make all the software infrastructure work with it, and reap the benefit of co-locality of the memory with the cores. Tae: And that’s how it’s optimized for LLMs? Exactly right. LLMs have these layers that do different parts of the work, and there are different ways to shard or parallelize them. That’s the trick. If you have a good way to separate it out and put it into the memory the core is operating on, with all cores operating in parallel, then you can be really efficient. That’s what we found. We found ways to do that with LLMs in general. Given any LLM, we can partition it out in this way so everything operates really efficiently. Memory bandwidth, compute, and networking are all balanced and all being utilized. So you get very high throughput as well as very low latency. That’s the key difference. A lot of older approaches generally bottlenecked at one point or another. They bottlenecked in prefill because there wasn’t enough compute, and then they bottlenecked in decode because there wasn’t enough memory bandwidth. So parts of the hardware sat there underutilized. Our goal was to get high utilization across the entire chip most of the time. So even though our peak marketing numbers for flops, memory, network, and bandwidth are comparable to other devices, we get this performance boost in real benchmarks because our utilization is very high. That comes from the mixture of the architecture with the software optimization we’ve done, using ML models to lay it out. Tae: Talk about your decision to choose a generalized architecture over a disaggregated one, and HBM over SRAM. Let’s talk about those two things separately. First, not disaggregating. I think disaggregation has a good story, but the fundamental issue is that when you deploy across a fleet in a data center, you have to commit ahead of time to how much of your hardware is going to be on prefill, how much on decode, how much on draft, or whatever else you have. Those are commitments that last years. You have to commit, the hardware goes in, and it takes a long time to back it out or change it. What we believe is that if you can make a fungible device that operates well across multiple parts of this domain space, you’re going to be in a better fleet management situation. You’ll get better utilization of your entire data center footprint, so the power going into your data center has a better chance of being fully utilized across the board. This takes the chip-level thinking about utilization up one level to the data center, where we want to use all the power in a data center all the time for very effective work. Disaggregation, although you can mitigate it with different workloads, fundamentally makes that more of a challenge. We want to remove that challenge and have full fungibility so we’re flexible. If you can do that, that’s great. What we found is that this architecture gives us a way to do that because of how we have the memory linked with the core. Once we saw that, we said, oh, yes, of course we should do that. We should not disaggregate, because we have this ability. It is a benefit, so reap that benefit. That’s what we thought there. On your second question about HBM versus SRAM. That fundamental trade-off is between bandwidth and capacity. You get a lot of bandwidth out of SRAM, but the capacity is so limited. At OpenAI, we talk about scale. Our models are really big, so you need a lot of capacity. So SRAM, although it’s a good solution, has a scalability problem. Right now, you can’t get more SRAM into a device than the process nodes from TSMC, Samsung, and Intel allow. There’s a limit there. HBM gives you a way to get beyond that in a meaningful way while still providing high bandwidth. Maybe I’ll answer the question that follows, which you haven’t asked yet. Memory hierarchy, memory design, and system design are super critical going forward. All ML accelerators need to think about that really hard. We think about that really hard. In our choices here, we balance capacity, latency, and bandwidth requirements to match them with compute and network interfaces, to get the high utilization and throughput we were talking about earlier. Tae: I was actually going to ask what you thought about the future of memory technology. There’s 3D-stacked DRAM. Obviously, you chose HBM4. Are you on the HBM train going forward? Is that a longer-term thing? We will always look at all the technologies, and we do. I’ll be honest with you. We do. At that point in time, HBM4 was the right choice technically. Moving forward, HBM is likely to be a component of our architecture in one way or another because of its benefits. Having said that, we are definitely looking at new advanced technologies, as I think all the chipmakers are. I think that’s an area where, again, this thinking in terms of utilization and reducing data movement is going to become super important. We hinted at it at the end of the Hot Chips presentation, and I reinforced that a little. We’re working on the second generation and the third generation. I’ll let you know we’ve even started working on the fourth generation. All of that is true, and where I think innovation is going to happen is at the intersection of what memory has to look like, what the layout is, and what technology we use there, along with the interconnect. As always, we’re going to ride as many flops as we can. Process technology will give us more flops, we’re going to ride that, and the other two areas are where we’ll innovate in order to keep up with it. Accelerator rack. Credit: OpenAI Tae: Jalapeno is meant to be, as I think I’ve heard you say, a high-volume chip. So how does OpenAI choose between using Nvidia, AMD, or its own chip? Is it just a matter of capacity? How do you decide what to use, where, and how much? It’s a capacity thing. We have so much demand for compute. Training is almost exclusively Nvidia, it’s all GPUs. So they’re a super important part of what we do. On the inference side, we have a huge appetite for more compute. If we can get that compute through our own chip, that’s great. If we can get it through Nvidia, that’s great. If we can get it through AMD or other vendors, that’s also great. To be honest, it all leads to better infrastructure. We want the best infrastructure we can get, so we’ll take compute wherever we can make use of it. Tae: One of the big achievements is getting the Jalapeno chip design to tapeout in nine months, which sounds incredible. Can you give specific examples of how AI enabled you to do this so quickly? Yeah. Okay. It helped us in many places. First of all, it helped us on the PPA [power, performance, and area] of the silicon. In particular, it was able to do explorations that a human would do, but a human would do them a lot slower and wouldn’t be able to do as many in the same time frame, because an AI can parallelize tens of thousands of runs. It can just do a lot. The way I think about it is that quite often a human expert uses the heuristics in their brain to figure out which explorations to make in order to get those optimizations. They have some idea of that. What I’m finding is that agents and AI models have been able to automate a lot of that heuristic exploration for us in a very intelligent way. They’re able to cover a much broader swath, and cover it more completely. They’re able to find points in the optimization space that we could probably get to ourselves as human engineers. It would just take us a lot longer. And there’s no guarantee we could get to the same hill-climb location that an AI agent can reach. They’re just so exhaustive and so good at this. That’s one thing. A similar concept applies to kernel optimization. The one thing I would say there is that, and I don’t mean this in a bad way, humans often have a certain way of thinking about a problem. One of the other advantages of agents is that they’ve seen so many different things and different approaches in their training data. They’re able to explore paths a human may not have thought of because of the range of that human’s experience and work history. So they’re able to find things where a human will say, oh, yeah, of course that works, but may not have thought of it initially. That’s where we’re seeing a lot of advantage. Because of their speed, their thoroughness, and their ability to find different solutions across a broad swath, they find things we just can’t get to as quickly or as well ourselves. We’re trying to take advantage of it everywhere. If you’ve been following recently, you’ll see that I took over the GPT-Infra effort here at OpenAI. That’s really about bringing the experience we had with the Jalapeno program to a broader range of applications within semiconductors, engineering, and infrastructure in general. Any improvement in infrastructure helps us, so we want to make that happen with our partners and help them do it better. Ultimately, we need to drive infrastructure costs down, because the world needs it, and it needs to be at the most sustainable point it can get to. Tae: You probably can’t say much about Gen 2 or Gen 3, but is this going to be an annual cadence going forward? I wouldn’t call it annual. It might be pragmatically close to annual. The key, to me, is that you have a certain overhead in deploying. You get a new chip, great. You talk about it, great. You benchmark it, great. But you have to actually deploy it in the data centers, and that takes a lot of effort and resources. So you’ve got to make sure each device is a marked improvement over the previous one in measurable metrics, such as performance per watt or intelligence per watt. You don’t want to do it every time you get a 2% or 3% increase. You want to do it when you get more major step functions. You want to be able to amortize the deployment, which is physical. It’s not like software, where you can roll it out as a release and the user suddenly sees a new model in the chooser, and that’s very simple. For us, rolling out into data centers is a lot more effort, so we’ve got to make each rollout incrementally worthwhile. An annual cadence is in that ballpark, but I’m not saying that’s going to be the guiding principle. We’re doing it based on whether we have a new device or a new system that gets us a step-function change in capabilities for our users. If yes, we should roll it out and then continue doing that. And that’s often tied to technology, as we were talking about earlier. Is there a technology change we’re going to adopt in a new device that would help us deliver that? That’s the analysis we’re doing all the time in our co-design, to see what we would need to bring in to make that happen and then plan it out from that perspective. Tae: I’m really interested in the topic of what makes a business culture effective and I loved your thoughts on small teams. First, how large is your team, if you can share a ballpark number? And why does a smaller team work better than a larger one? The team is about 100 people, but those 100 people cover everything. Chip design, verification, physical design, packaging, thermals, system design, board design, rack design, supply chain, software, and firmware. It’s a pretty broad space. One hundred people for that is very small. The chip design team itself is tiny. It’s only a small fraction of that. This works because small teams, in our experience, move faster. If you have a set of people who are experts in their space, with the tools backing them up in terms of agents and ML, they’re able to make decisions very rapidly and optimize. One of the things I’ve said before as an engineering manager is that your speed of innovation depends on your iteration loop. If you iterate on something, get the result, measure it, figure out what the changes are, and iterate again, you’re going to go much faster. A team that can iterate in an hour will outpace a team that takes a week to iterate. So it’s the number of iterations that really matters. This is the business side of it. I believe in fast iterations, and I think Sam and the company also believe in fast iterations. They iterate, deploy, and modify. In our engineering practices, this small team really helps with that. Iterate fast, measure, see what needs to improve, and iterate again. That’s how we move very quickly from “Here’s the goal” to “Here’s an idea. This part worked. This part didn’t work. What should we modify? Let’s keep going.” You’ll be surprised sometimes, and I’m surprised, how fast you can get to a point you never thought would have been possible had you not taken that first step, that first iteration. There’s an old saying that a journey of a thousand miles begins with a single step. You’ve got to start with that first step of iteration, and from there you’re able to walk the journey. You may not see exactly where that journey is going to end up, but you can’t get there unless you take the first few steps. That opens up your knowledge base and lets you see what else is coming over the horizon that you couldn’t see from the starting point. Tae: Every few weeks, I have to deal with a misleading media narrative saying the AI theme is over, AI is good enough, free open-source models are going to overtake the frontier, and you don’t need the frontier anymore. In terms of what you’re seeing inside OpenAI and looking out over the next year or two, how early are we in AI? What are the potential AI capabilities over the next two to three years? I assume you disagree with these negative AI takes. Yes, I do. The way I like to think about it is that we as humans often have difficulty understanding what exponentially increasing capability actually translates to. What we’re seeing here is definitely an exponential increase in capabilities. We’re seeing that just in terms of models coming out every few months that are 10x better than they were before. I think most people think about capabilities increasing more linearly, because that’s the way humans think. So it’s harder to grasp that the model we’re going to get a year from now is going to make today’s models look very primitive and not as intelligent. So where is this going to happen? I believe it has to happen at the forefront of research. And I believe we’re at a point in time where, because the capabilities are so strong, you need researchers actively working on guidance, safety, and guardrails, because the capabilities have gotten so good. You need to put that kind of effort in. Is it going to happen in open source? Maybe, but to be honest, I don’t see much of it happening in open source. They’re mostly trying to get more capabilities. But I do see it happening in the labs. I see it happening here. I see it happening at Google. I see it happening at Anthropic. I see it happening elsewhere. And I think that’s a necessary thing. This is going to be a super important capability for humans as a whole, but we have to do it right. We have to move fast, but we have to do it right. It has to be done in a very thoughtful and deliberate way, by people who care about that, which I definitely see happening within OpenAI. That’s the key thing I see happening. So I push back on that one. I don’t think we’re anywhere close [to the end of AI advances]. It feels like we’re a lot closer than we were, but I feel there’s a lot more intelligence still to come. It’s really on us as a community, as a species, to make sure this is done right. This is a critical moment in time. Thanks for your time.