Apple Silicon and macOS VMs: 11–16× Faster LLM Inference with Llama.cpp

engzaanin•about 1 hour ago

That makes sense. The title initially sounded like a general llama.cpp speedup on Apple Silicon, but if the improvement comes from fixing kernel selection inside Virtualization.framework VMs, that distinction is pretty important.

simonw•about 1 hour ago

It looks to me like this won't speed up llama.cpp for everyone, just for users running it in this particular kind of Virtualization.framework VM.

The fix here works around a problem where the VM was causing llama.cpp to select the wrong kernels.

frabonacci•about 1 hour ago

> this won't speed up llama.cpp for everyone, just for users running it in this particular kind of Virtualization.framework VM.

correct. these figures apply to llama.cpp inside the macOS guest configuration we tested. Lume is the VM frontend we used, while Apple's Virtualization.framework provides the virtual GPU. bare-metal llama.cpp is unaffected.

> The fix here works around a problem where the VM was causing llama.cpp to select the wrong kernels.

mostly, with one nuance: llama.cpp is selecting the correct kernels for the capability answers it receives. the stock guest reports an older Apple GPU family and a 32 KB threadgroup memory limit, so llama.cpp chooses slower kernels. Our process-scoped layer reports the tested Apple 9 and 64 KB values while allowing llama.cpp to select newer paths that the paravirtual GPU successfully execute

the layer itself though works at the Metal API boundary, independently of llama.cpp. other Metal compute and graphics apps now may select newer paths from the same capability answers, although this is still preliminary and each app needs separate testing. for example, MLX-LM stayed flat in our tests

historically related limitations have been coming up across Apple Silicon VM frontends for a while e.g. Tart tracked MPS/GPU support back in 2023: - https://github.com/openai/tart/issues/501 - https://github.com/openai/tart/issues/1032

UTM also has related cases where apps detect the Apple paravirtual Metal device but falls back to software rendering: https://github.com/utmapp/UTM/issues/7671

thehamkercat•about 2 hours ago

> 11.08× faster and generated tokens 16.36× faster than the same workload in the same stock VM.

So this was the comparison, for me the title was a bit confusing

frabonacci•about 1 hour ago

yeah fair point. it's always tricky to get the whole idea across within HN's title limit. tldr: we ran the same workload in the same Lume macOS VM on the same Apple Silicon host, first with stock Metal capability reporting and then with our process-scoped dynamic library. The 11.08x figure is prompt processing, while 16.36x is token generation. the mechanism technically extends to graphics workloads too but these figures are specifically from llama.cpp

luciana1u•20 minutes ago

my whole setup is buy more RAM, run it on CPU, and tell myself the GPU is just a personality trait I'm working on.

azinman2•about 1 hour ago

I don’t understand what Apple 1-9 are. At first I thought it was M series chips but there is no M9 (yet)

niklasbuschmann•about 1 hour ago

https://developer.apple.com/documentation/metal/mtlgpufamily

wtallis•about 1 hour ago

So those generation numbers aren't really anchored to Apple's hardware designs. It's just counting from when Apple introduced the Metal API, and the first several generations were when the GPU cores Apple was using were still nominally PowerVR designs.

shay_ker•about 1 hour ago

I recall there was another YC startup that was working on Mac-specific ML optimizations for local inference (and perhaps fine-tuning).

I wonder if their work is related?

frabonacci•about 1 hour ago

RunAnywhere or Conifer?

woadwarrior01•about 1 hour ago

The Claudish in the blogpost makes it really hard to ready. Also, TinyLlama 1.1B lol.

Apple Silicon and macOS VMs: 11–16× Faster LLM Inference with Llama.cpp

⚡ Community Insights

Discussion (20 Comments)Read Original on HackerNews