CUDA for AMD on Windows

On the SDK front, you tried just having an agent reimplement the model you're interested in and just use the weights? I've taken to treating off the shelf implementations as reference implementations anyway, because I can often squeeze out significantly better performance for my configuration and use case by having Codex hammer at it for a few hours.

The problem with the open standards is that their dev UX is absolutely horrible. You can't neglect usability, and then be surprised that there are no users.

Commercially it's better to have AMD support CUDA which can help break Nvidia soft monopoly.

I would too, but sadly that's Khronos' job to organize, and they've had trouble getting American vendors to work together.

It's likely that CUDA will continue dominating until they put aside their differences. The current MLX/MPS/ROCm ecosystems are too fractured to threaten Nvidia.

it will only get worse now when hughingface was bough by nvidia, not immediately, but in span of year or years...

It's impossible to have an open standard. Hardware accelerators are nothing alike and have different perf characteristics. Each kernel is tuned to the hardware. The idea of writing a performant kernel in opencl is a fantasy.

Source: worked at a bunch of accelerator companies in the kernels or equivalent team. They're nothing alike.

trivial to translate (or transpile) - okay. trivial to understand the result - not so much. trivial to then evolve it - hm... perhaps a different story. still, it seems very likely now, that such "quick rewrites" are viable, not sure if an open approach to them is viable. a newly born open project that was LLM-derived, and not by a credible author, which spans hundreds of files no human eye has ever looked at, can only work for a closed organization, but will never be trusted by the general audience... just like that.

> When it becomes trivial to translate CUDA/PTX to HIP,...

ZLUDA is already doing that, no?

yeah yeah, "when" an often keyword with AI it seems. As Mr. E. Nigma put it - what always comes but never arrives? Meanwhile the moat deepens and it's build on inertia and laziness and Nvidia knows this really REALLY well.

i swear people who are outsiders here have only clickbait takes; if you've never had to ship GPU code professionally you should just not comment on these things.

the source language has never been the moat. Nvidia sells to hyperscalers. Hyperscalers have armies of kernel authors who have no issue translating shaders by hand (or now with claude). Nvidia's moat is (and will remain for the foreseeable future) the entire stack. you cannot fathom the pain and misery of working on literally any other stack. if you've never debugged a GPU synchronization error or kernel panic due to some GPU firmware bug or fought absolute shit profilers hunting for perf you really have no idea what you're talking about.

EDIT: i can't believe this really requires saying but graphics and compute are not the same domain at all. if you work in graphics for GPU but not compute then you are still way out of your depth commenting. to wit: graphics people do not (and cannot) write CUDA kernels/shaders.

What models can it run?

RDNA1 isn't good for a whole lot, even flagship RDNA2 cards are a stretch for many things. The lack of WMMA/matrix multiply/BF16 is too severe of a penalty.

The FP16 throughput on RDNA1 is both shader reliant and requires everything to be packed first. Even with 2 or 4 or 1000 cards, you would be consuming all of the available memory and memory bandwidth just packing and unpacking values, and if you really want to dump a hundred billion tokens into making it work anyways, you're only going to find out that even if you bother to sit there ferrying packed values to ram or disk before then issuing the instructions, paying that already severe penalty again when the values then have to be unpacked is so steep of a cost that the 256 BF16 flops/cu/clock's effective throughput is outright lower than simply doing it on a Zen 2 processor. You also don't have INT8 (or really INT4) on RDNA1 so the other RNS/CRT tricks aren't viable.

Sadly RDNA1's VCN2 also lacks actually good x264 bframe encoding support, or even P010 for 10 bit color, so what I'm saying is you should sell them. Used Radeon VII's are like $260, you'll go a lot further with those especially if you throw in a 7900XTX, and then augment that further with a 9070 CRE (you only want it for its int8 cores), and of course 128GB of ram.

E: And sure, that's 3, or ideally 4 GPUs, and a good bit of extra work. But that gets you up to more than halfway to the naive performance of a $15,000 MI300x in a surprising amount of cases, with additional strengths that it lacks. For far less than half of the cost

(As a side note, I love the name ZLUDA; it very aptly means "delusion" or "deception" in Polish.)