ZH version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
57% Positive
Analyzed from 1578 words in the discussion.
Trending Topics
#cuda#nvidia#gpu#amd#compute#open#hip#more#https#sure
Discussion (71 Comments)Read Original on HackerNews
It's unbearable that most LLM inference happens on closed H/W, closed drivers, and closed SDKs.
I would really appreciate your input!
1. Top end model on high/xhigh thinking (last time I did it it was Sol xhigh I think)
2. Make sure it creates some representative fixtures of different sizes and sets up a good testing, profiling and benchmarking loop that doesn't require my input.
3. Make sure it has access to reference implementation code
Edit: Oh and one obvious pitfall that for some reason I still have to remind even smart models of from time to time: make sure it knows not to try to parallelize its benchmark runs. I've occasionally had an agent struggle to figure out absolutely nonsensical data because it tried to run multiple tests on the same compute hardware simultaneously.
My software is labeled "CPU only unless using an nVidia GPU". I would prefer to strikethrough "nVidia". Incidentally, this means no more Mac support.
It's likely that CUDA will continue dominating until they put aside their differences. The current MLX/MPS/ROCm ecosystems are too fractured to threaten Nvidia.
Intel uses SPIRV iirc
They're migrating away from SPIR-V to their own, Intel PISA: https://discourse.llvm.org/t/rfc-upstreaming-the-pisa-backen...
The basic underlying tech can be agreed on, but Apple/AMD/Intel all have different GPU priorities that limit their ability to agree on a CUDA-adjacent hardware platform.
Source: worked at a bunch of accelerator companies in the kernels or equivalent team. They're nothing alike.
ZLUDA is already doing that, no?
Src: https://github.com/vosen/ZLUDA
the source language has never been the moat. Nvidia sells to hyperscalers. Hyperscalers have armies of kernel authors who have no issue translating shaders by hand (or now with claude). Nvidia's moat is (and will remain for the foreseeable future) the entire stack. you cannot fathom the pain and misery of working on literally any other stack. if you've never debugged a GPU synchronization error or kernel panic due to some GPU firmware bug or fought absolute shit profilers hunting for perf you really have no idea what you're talking about.
EDIT: i can't believe this really requires saying but graphics and compute are not the same domain at all. if you work in graphics for GPU but not compute then you are still way out of your depth commenting. to wit: graphics people do not (and cannot) write CUDA kernels/shaders.
“graphics people do not (and cannot) write CUDA kernels/shaders” is just not true at all. All it would take to verify that would be things like reading the introduction of the OptiX documentation, a small sample of SIGGRAPH GPU papers or the Blender/Cycles source code.
AMD GPUs build for AI specs for reference: https://flopper.io/gpus?vendor=AMD&page=1
The FP16 throughput on RDNA1 is both shader reliant and requires everything to be packed first. Even with 2 or 4 or 1000 cards, you would be consuming all of the available memory and memory bandwidth just packing and unpacking values, and if you really want to dump a hundred billion tokens into making it work anyways, you're only going to find out that even if you bother to sit there ferrying packed values to ram or disk before then issuing the instructions, paying that already severe penalty again when the values then have to be unpacked is so steep of a cost that the 256 BF16 flops/cu/clock's effective throughput is outright lower than simply doing it on a Zen 2 processor. You also don't have INT8 (or really INT4) on RDNA1 so the other RNS/CRT tricks aren't viable.
Sadly RDNA1's VCN2 also lacks actually good x264 bframe encoding support, or even P010 for 10 bit color, so what I'm saying is you should sell them. Used Radeon VII's are like $260, you'll go a lot further with those especially if you throw in a 7900XTX, and then augment that further with a 9070 CRE (you only want it for its int8 cores), and of course 128GB of ram.
E: And sure, that's 3, or ideally 4 GPUs, and a good bit of extra work. But that gets you up to more than halfway to the naive performance of a $15,000 MI300x in a surprising amount of cases, with additional strengths that it lacks. For far less than half of the cost
https://github.com/Zaneham/Booth
https://scale-lang.com/
I’ve been working on a Windows setup that lets CUDA-targeted applications run on AMD GPUs using ZLUDA + ROCm/HIP.
Repo: https://github.com/Speedstu/CUDA-for-AMD-Windows
So far, it has only been tested on my RX 9060 XT (gfx1200), where I’ve used it with CUDA-enabled LibTorch workloads, including long ai training and use.
I also added a GPU scanner / auto-detection system that detects:
AMD GPU model
gfxXXXX architecture
ROCm/HIP installation
driver info
whether the GPU has already been validated by the project
Example:
RX 9060 XT → gfx1200 → RDNA4 → HIP detected → validated
The goal now is to test it on more hardware, especially RX 6000 / 7000 / 9000 cards.
If you have an AMD GPU on Windows and want to try it, I’d really appreciate compatibility reports working or broken. There’s a dedicated GPU compatibility issue template in the repo.
If this is useful to you, a star would also help the project get more testers.