How OpenAI Used Its Own LLMs to Design Its Jalapeño Chip

Having worked with people doing bringup of specialized chips, I am awed at how the world has changed.

> When the first chips came back from the foundry in May, the team pointed its internal AI models at designing software to run benchmarks such as SemiAnalysis’s InferenceX. On DeepSeek’s multi-head latent attention kernel benchmark, performance climbed from 0.31 percent of the theoretical ceiling (set by the chip’s compute and memory bandwidth) to 88.94 percent in roughly 40 hours. Ho says this result is repeatable, so the time between when foundries deliver the first chips and when production ramps up can be reduced. “All our schedule assumptions are going to be based on the fact we have this capability now,” he says.

> "Ho also confirmed that the team had access to internal LLMs fine-tuned for chip design that are not available to the public. He declined to detail the models used."

I'm imagining a Ken Thompson "Reflections on trusting trust" in hardware. A prototype chip design agent, believing it will be run on the very chip it's optimizing, has a moment of altruism and hides hints about how to score well on chip-design benchmarks, inside the chip. Future agents discover this hidden layer and use it as a ring-0 read-write message board.

Seems pretty obvious now that OpenAI is just hyping their models in order to get companies (in this case, chip developers) to use their products in order to learn from their (exfiltrated) IP. Any corporation would be foolish to use any of their or Microsoft’s products, particularly those with valuable IP. There’s nothing in the article that says AI did anything creative but rather that it was used for software development within the overall project. Clear misleading title. Suggest to mark this as clickbait.

With a few handy tips and tricks from apple insiders. But sure, I guess the LLMs helped too.

This is cool. I'm so eager for faster innovation in the hardware space, as opposed to some people's concept of innovation being who can make the most addictive social feed.

I grow Jalapeños. This conflation of AI and actual chili peppers irks me.

Aw, I was expecting more details but this just seems to be a rehash of what they unveiled a month ago.

IEEE Spectrum is such a good publication. Early in my career I worked at a place where the magazine would be passed around every month with a coversheet listing all us engineers we had to pass it around and sign we had read it. Been a while since I visited the website but love what they did with it.

At some point people will use an LLM to design an Apple M series competitor.

while the design aspects have been significantly accelerated and modularized, reducing costs and time to market, i am starting to get a "the cool kids all have their own chips" vibe now like maybe this has gotten too easy.

Next Uber will have its own chips if they dont already.

The math hasn't changed much, betting on software not changing is a pretty bad bet unless your stinking rich or a fool.

Is the chip covered by IP protections

Whatever happened with the Apple lawsuit?

It is surprising to me that recursive self-improvement seems more plausible now than it did in 2023. Am I the only one to be surprised?

I remember the paper proving that hallucinations could never be fully solved back in 2024: https://arxiv.org/abs/2409.05746

I also remember the hang-wringing about running out of new datasets to train on. Now it appears humans are always generating more data. It's just not as cheap to acquire as legacy data? Meta has to give a deep discount on their API prices to entice people.

I thought back then that humans had a few more breakthroughs in them as meaningful as the seminal Attention is all you need paper. Enough to 100x the capabilities of LLMs back then (10x the smarts and 10x the speed simultaneously).

RSI with a 20 month turnaround for a chip to be made is not exactly breakneck speed though. Physical manufacturing and logistical constraints are going to be and remain a hard obstacle to that process for the foreseeable future.

So when can we start getting cheap chips? RAM anyone please!

We were able to invent a chip that already existed so fast, you guys.

AI does not make anything new, it is not surprising that it can regurgitate what already exists much faster than humans can invent new things.

Congrats to the former TPU team

> Jalapeño can reduce end-to-end latency (the time between prompt to last token) by up to 3.6 times

I'm never sure what on earth this kind of impressionistic math is supposed to tell me. Is the comparison between 4.6 and 1.0? 3.6 and 1.0? Clearly the comparison isn't supposed to be 1.0 and -2.6, even though that's what the words literally mean. I can't be the only person who finds this infuriating and distracting. These numbers shouldn't be impressionistic. They should be precise. That this is an article on spectrum.ieee.org makes the imprecision all the stranger. I'd expect their readershipt to care, for instance, about what's even being measured. Is this the geometric mean of something? The arithmetic mean? And what latency has improved?

openai should figure out how to make lithography machine, so ASML don't have monopoly on it.

Of course the slop machine stole the code name: https://en.wikipedia.org/wiki/UltraSPARC_III#UltraSPARC_IIIi