Popping the GPU Bubble

Thank you for the kind words. We will write and share more of these.

> Similar to compiler engineers before.

I guess the difference here being that we have ample compiler literature and practically know 99% of all there is to know about compilers that exist in the wild vs this new field.

Until we’ve gathered and agreed on a few “dragon books” for LLMs and have explored all there is to LLMs, you’re probably right - know-how will be with the practitioners and in source code until it’s distilled (pun intended).

Most industries are like that.

Gentle reminder that while most money is spent on LLM inference, the vast majority of useful AI use is in fact not LLMs. Also, more and more work is poured into making small models. One thing I like about the whole export controls saga is that people are finding creative ways to squeeze performance out of these devices as witnessed in this post. But, if you then look at solutions like vLLM, vLLM will just fill whatever VRAM is available, no matter the context size, or the model size. So then you have two things to worry about:

First, where do you know exactly what the optimal VRAM assignment per model, per context size is, which seems to be currently based purely on experience and second how do you make sure that only that amount is available to your infra/containers, which is being handled by DRA and stuff like https://project-hami.io

While only tangentially related to the blog post here. The title is picked in such a way that I couldn't help, but put the shameless plug here. When he wrote popping the bubble, I thought we're talking about devices and reducing NVIDIA dependency, but this seems very focused on Cuda.

Disclaimer: I work with Dynamia.ai, the founders of which created HAMi.

Appreciate you saying the blog was nice. Not sure what you mean by "CODEX fingerprints", but I'll engage with the other points. We work on small models, and our customers want real-time inference on modern GPUs. The sub-title says "near-realtime VLM inference". 20-30ms forward passes are a non-starter for these workloads.

If you scroll down to the section titled "A cost model for the bubble", you will find both benchmark results and us saying, "you get back anywhere from a few percent to a third; more the faster your accelerator/model is".

It appears to be a real term? https://docs.vulkan.org/tutorial/latest/Synchronization/Asyn...

Very odd, but perhaps more familiar to graphics programmers? I will say I'd probably call it a stall, which is exactly what the Vulkan docs call it moments later, so :shrug:

Pretty sure that would be "[GPU performance] bottlenecked [by the CPU]" in most common terms.

while the title is misreading, when reading GPU profiling data, we do call these bubbles - where the GPU _could_ do something, but it's idle.

any time your GPU is idle = you are losing $$$ = your TCO is going up. you don't want that.

It's very common to call it a GPU bubble in gamedev, though not strictly for CPU induced bubbles.

I saw it in literature on cpu pipelines in quotes, never without.

Yes, the title seems off - I also thought I am going to be reading about the AI/pricing bubble.

The term I would use would be “underutilised”

I'm disappointed with the commentary here. "GPU bubble" is an industry standard term, and literally how I would describe this to my colleagues in the industry. Look for example at the second slide here https://media.steampowered.com/apps/valve/2015/Alex_Vlachos_...

This is what people in the field call it. I'm sorry you're offended.

Yeah the title is obviously clickbait.