Ask HN: Who is using FPGA for ML inference?

ssoftwarewright about 4 hours ago 5 comments

DE version is available. Content is displayed in original English for accuracy.

With RAM price inflation, I wonder if FPGAs can be used to offload inference processing without keeping weights in RAM? The available RAM would be for activations, KV Cache, context but not static weights. Weights could be streamed from disk. This approach is not for tokens/second but throughput at a lower cost. Possibly better answers/kHh? I've started researching this, but wonder if others have considered/tried this?

⚡ Community Insights

Discussion Sentiment

50% Positive

Analyzed from 176 words in the discussion.

Discussion (5 Comments)Read Original on HackerNews

wmf•about 3 hours ago

FPGAs are far more expensive than GPU+DRAM, even at today's inflated prices.

softwarewright•about 2 hours ago

Tang nano FPGAs are around $10, the idea being to couple with a $25 ARM or RISC-V dev board and offload the math from a streaming store. Coupling this with an older 8GB VRAM board can theoretically run a model whose weights do not fit. The weights get repeatedly run through the MCU/FPGA and the VRAM is used for KV Cache and context. If you connected several via USB and streamed a MoE model expert per FPGA, you could achieve parallelism. Won't be fast, but, many requests could be processed in parallel, as each request shares the large static weights.

The reason for this is, model weights do not need to be randomly accessed. So why store them in expensive RAM.

Cerebras and Qrok seem to be using a very different approach than NVIDIA to get orders of magnitudes speed ups. I'm trying to explore other alternative approaches.

wmf•about 1 hour ago

I predict that the FPGA adds no value in this scenario. Just process inference on the CPU.

pugfugly•about 2 hours ago

Train circuits, not weights; then use an FPGA

mathisfun123•about 2 hours ago

No one because FPGAs are 10-20x less dense than ASICs actually designed for purpose. The tooling is also complete shit. Signed someone that did (part of) their PhD on this topic.