The efficient frontier of LLM inference

pphilipkiely about 7 hours ago 25 commentsRead Article on baseten.co

ES version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

93% Positive

Analyzed from 834 words in the discussion.

Discussion (25 Comments)Read Original on HackerNews

jumploops•about 3 hours ago

> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses.

As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack.

Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design.

Then in the mid-2000s the Speculator[0] paper brought that concept to distributed systems, which we’re still seeing work on[1][2].

Everything old is new again (:

[0]https://www.cs.princeton.edu/courses/archive/fall07/cos518/p...

[1]https://www.usenix.org/system/files/osdi25-shen-weihai.pdf

[2] https://www.microsoft.com/en-us/research/publication/distrib...

bessbd•about 1 hour ago

> guessing which tokens a model might generate

A transformer that generates tokens based on pre-training? We could call it a GPT for short.

freakynit•about 3 hours ago

Can we expect similar issues such as spectre and meltdown that intel experienced with speculative execution.. but, in the form of prompt injection/poisoning?

rf15•about 2 hours ago

Ok, I'll bite: no, considering these are very different domains and you don't get system access by getting the wrong speculative branch for your next text token, you just get a slightly different (but probably still related enough) text.

mathisfun123•about 2 hours ago

> optimizations applied at different levels of the stack

That's because it's just "guess and check" not some deep universal insight.

wongarsu•17 minutes ago

It's closer to "we can make this highly parallel for not that much cost, but we struggle to use that concurrency. So what if we just guess what the next step is going to be? If we are right we get a big speedup, if we are wrong we just throw that work away". Which I would classify as a notable insight. Doing work that you are 50% certain is useless is not the most obvious thing

jumploops•about 1 hour ago

Simple optimizations are often the best :)

bit_rot73•8 minutes ago

My RTX 3090 is still laughing at my attempts to run 70B models efficiently.

clem_rw•27 minutes ago

My daily struggle is trying to make a 7B model respond in under 500ms without breaking the bank. This hits home.

ttoinou•about 6 hours ago

   Inference techniques either move a deployment along the latency–throughput frontier or push the entire frontier out, creating more efficiency to allocate.

This is a tautology. You can say that with anything. Gastronomy techniques will make a previous recipe better, or create a new recipe better than others, or a mix of both.

Ifkaluva•about 4 hours ago

The point is to classify them into two kinds. The kind that shifts the frontier is more powerful, since improves capabilities without incurring tradeoffs.

philipkiely•about 4 hours ago

I also wrote this as somewhat of a defense of the techniques that don't move the frontier -- there is a lot of value in being able to pick a point on the curve.

brrrrrm•about 7 hours ago

this is a nice and concise writeup. what's striking to me is that these techniques really have not changed in /years/. sure, precision has become slightly lower, spec decoding acceptance has gotten slightly better and the complexity of parallelism is trickier with mixture of experts. but no new concepts in a very long time!

the absolute most impactful improvements for inference comes at architecture design time. I firmly believe everyone who cares about impacting model efficiency should look there

philipkiely•about 6 hours ago

I think the biggest net new recent technique is P/D disaggregation. And that spec dec is very different now especially post DSpark/DFlash.

But overall yes the fundamentals of LLM performance optimization have been remarkably stable over the last few years.

brrrrrm•about 5 hours ago

perhaps its unfair to say this in hindsight, but it's a fairly straightforward application of little's law that's been around for some time

https://arxiv.org/html/2401.09670v2

censor25•about 2 hours ago

Nice read. I was wondering what can one do to get into inference engineering as simple theoretical knowledge is not sufficient and switching profiles is tough for someone with years of experience.

arjie•about 3 hours ago

You know what I'm curious about? Whether you have brand guidelines inside the company, a Claude skillset, or the blog post author makes the charts in line with the brand colours and so on.

philipkiely•about 2 hours ago

I draw my diagrams on notecards and send them to our designer who brings them to life.

The images start out looking like this: https://philipkiely.com/images/blogs/how-to-write-a-book/des...

arjie•about 2 hours ago

Makes perfect sense. The classic way!

qingcharles•about 2 hours ago

> A model is a “frontier model” if it offers the highest degree of intelligence at a given cost or size.

I would define a "frontier model" as offering the highest degree of intelligence at any cost, or without regard to cost. The frontier today is clearly Fable/Mythos, with the "efficient frontier" at Opus/Sol.

orangeboats•about 1 hour ago

Your partial quote is quite misleading. The article obviously talks about "efficient frontier", not "intelligent frontier".

>In the AI industry, we borrowed the term “efficient frontier” from economists. We use it to talk about managing tradeoffs, most often the tradeoff between cost and capabilities for models. A model is a “frontier model” if it offers the highest degree of intelligence at a given cost or size.

datadrivenangel•about 6 hours ago

The author does not deeply mention that quality/intelligence is a third dimension here in addition to throughput and latency, and the frontier is jagged so quality and intelligence require bespoke benchmarks to evaluate tradeoffs for speed and cost.

philipkiely•about 6 hours ago

These are both good points that I attempted to cover, quotes:

> In practice, the efficient frontier is very jagged. Rather than a smooth, continuous line between outcomes, small changes can have big impacts. These cutoff points are often unintuitive and must be discovered empirically through sweeps.

> However, quantization introduces a new set of tradeoffs between quality and serving efficiency. This is a particularly jagged frontier, where a large degree of improvement to serving efficiency is possible with little-to-no reduction in model quality, especially when using microscaling floating-point number formats like MXFP4 and NVFP4.

Would appreciate ideas on how to explain in greater depth

calclavia•about 6 hours ago

good recap on the recent inference techniques!

fsckboy•about 3 hours ago

"the efficient frontier" is an important landmark of (investment) portfolio theory. It proves/explains/illustrates how you can combine selections from a diffuse cloud of individual investments and still land on a frontier that is better than any of your individual choices. It's the entire basis of "diversify your portfolio".

The efficient frontier of LLM inference is a line, not a frontier.

this is a frontier:

https://upload.wikimedia.org/wikipedia/commons/e/e1/Markowit...

no matter how good is something a smart person writes down, a pleb will come along and try to hang on its coattails.

If you want to steal an idea for this, steal indifference curves, they'd make more sense.