Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

268

ggitpusher42 about 2 hours ago 71 commentsRead Article on github.com

FR version is available. Content is displayed in original English for accuracy.

Hi HN,

I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal.

I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory.

The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are included.

The trick is to keep the shared part of the model and the KV cache in RAM, then stream only the routed experts needed for each token from SSD. An SSD is way slower than RAM, so the runtime uses a small expert cache and bounded parallel `pread`. While those reads are in flight, the GPU runs the shared part of the layer.

I ran more than 100 experiments. Most didn’t work. A few got me here. The experiments are described in the GitHub repo.

It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.

I also added an experimental OpenAI-compatible local server. It supports streaming and tool calls, and reuses one prompt prefix from the KV cache.

Try it! The Mac app is easy to install. On the first run, it will download 15 GB of weights from Hugging Face. The model is surprisingly capable.

I would love any kind of feedback!

⚡ Community Insights

Discussion Sentiment

88% Positive

Analyzed from 1966 words in the discussion.

Discussion (71 Comments)Read Original on HackerNews

xenonite•41 minutes ago

With my M1 MBA, I am still on macOS 15. To compile it, just remove the two lines with

  opts.languageVersion = .version4_0

or surround them with

  if #available(macOS 26.0, *) {
    opts.languageVersion = .version4_0
  }

You'll miss out on a prefill speedup of 2.4x (as it yields 11.24x faster attention), according to the git comments, but it works. (On the 8-GPU-core MBA M1, I get 5-6 tok/s.)

gitpusher42•35 minutes ago

Thank you! That’s useful. I might try lowering the minimum version later. The 2.4x prefill improvement will only work on the apple10 GPU family. The M1 uses apple7 as I remember

tredre3•about 2 hours ago

I'm curious how your project compares to plain mmap!

Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled).

It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.

gitpusher42•about 1 hour ago

My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread`

With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work

And common weights still use mmap for simplicity

So, I believe llama.cpp might run it under 2gb, but I assume it will be slower

nijave•13 minutes ago

Any idea if madvise helps? Admittedly I have very limited experience and only on Linux

a-dub•39 minutes ago

for a given expert, do you have a sense for what the spatiotemporal access pattern looks like?

Catloafdev•about 1 hour ago

Ya I'd be interested to see a comparison of using llamacpp with ssd offloading to compare real speeds.

owaislone•about 1 hour ago

Exciting! Maybe techniques like these can enable systems with 30-60GB memory and very fast SSDs of the future run very large models hopefully.

gitpusher42•about 1 hour ago

Yeah! Check the Colibri and Flash-MoE projects. They’re already doing that.

https://github.com/danveloper/flash-moe https://github.com/JustVugg/colibri

hacklas•16 minutes ago

How large?

With 64 GB of unified memory, you should be able to run a DeepSeek V4 Flash quantisation at 7–10 t/s, for example with: https://github.com/antirez/ds4 or https://github.com/steadfastgaze/MoEspresso (my engine).

The routed experts needed for the next tokens that are not already in memory need to be read from the SSD, so the speed becomes SSD reading bound and the larger the memory, the faster the inference.

mandeepj•12 minutes ago

> 7–10 t/s

Maybe use it for overnight batch work! Hopefully, you aren’t suggesting it using for realtime conversations!

mmastrac•about 1 hour ago

I have a project that's almost ready to run DiffusionGemma as well. The two project might potentially work well together. I'm getting ~20tok/s on a 36GB M3 and there's strong possibility we might be able to crib faster kernels from each other.

Feel free to reach out.

(currently at https://github.com/mmastrac/diffgemma but not in a releasable state yet)

eamag•about 1 hour ago

Cool project! I looked into it recently and thought that running diffusion models locally doesn't really make sense: https://eamag.me/2026/why-parallel-diffusion-llms-are-slow-o...

What are your thoughts on this?

gitpusher42•about 1 hour ago

It is super cool! Diffusion Gemma was released around the middle of my project, and I seriously considered switching to it. But I decided to finish the project as it was.

I believe it would be a perfect match!

Feel free to use any parts of my project or drop me a message. There’s my LinkedIn link at the end of the readme. Or I will drop you a message later!

addaon•about 2 hours ago

> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.

Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...

GeekyBear•about 2 hours ago

The M5 SSD's performance uplift was fairly substantial, even when compared to the prior generation.

> In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift.

https://www.tomshardware.com/laptops/macbooks/m5-macbook-pro...

ambicapter•about 1 hour ago

Extremely impressive.

Barbing•11 minutes ago

Not sure Data (Star Trek TNG) could read that fast.

“Accessing”

wongarsu•about 2 hours ago

My suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads

If that's true, inference speed would be even lower if you have only 2GB total, including OS caches

Schiendelman•about 1 hour ago

Not only is it older, so Pro:Pro it would have much slower SSD, but doesn't the Air also have slower SSD than the Pro in the same generation? And maybe narrower memory bandwidth?

tracker1•about 1 hour ago

IIRC, depends on the SSD size.. larger sizes had 2x the bandwidth, so it depends. That's combined/offset with the M5 improvements even further.

afzalive•about 2 hours ago

The M5 MBP has 24GB of RAM, more context in RAM perhaps?

gitpusher42•about 1 hour ago

The process stays at around 2 GB with 16 slots and a 4K context on both the M5 and M2. But yeah, Apple might be doing some magic under the hood

petu•about 1 hour ago

Unused RAM is wasted RAM. So not really Apple magic, about every OS uses "free" memory as disk cache.

Try to leave only a gigabyte or two free, speed likely would drop dramatically.

Edit: or do some calculation / logging of experts read speed, to see if it's faster than SSD spec.

luciana1u•26 minutes ago

26B in 2GB is the engineering equivalent of fitting your entire apartment into a storage unit and still having room to pace

tpurves•34 minutes ago

Okay this tidbit is interesting to me "5–6 tok/s M2 -> 31–35 tok/s on an M5 Pro". So where will be in just another gen or two?

my impression right now is that M5 gen is on the cusp of practicality for local inference.

If techniques like OPs here, start to make the RAM situation more amenable, by the time we get to M6 or M7 (or AMD's equiv next gen APUs on TSMC N2 nodes), local AI could be ready to go much more mainstream.

Aurornis•14 minutes ago

I have an M5 128GB. Being on the cusp of practical is a good description. It will run, but prefill and token gen are still slow relative to my consumer GPU box.

It also gets very hot. If you’ve never heard the fans on Apple Silicon really spin up, it could surprise you. Makes the full GPU setup feel quiet by comparison.

I think after the hardware market calms down the ticket is going to be a light laptop with a second dedicated inference server on the network.

hatthew•19 minutes ago

My assumption is that the difference is 90% from more memory. I'm making several assumptions because nothing here looks groundbreaking so I don't care to dig deeper, but the model + KV cache definitely cannot fit in memory on the 8GB machine, but probably can on the 24GB machine—or can at least get close. Assuming that this benchmark makes use of that, skipping SSD streaming will speed things up massively (I would have guessed much higher than the reported 6x speedup).

trollbridge•23 minutes ago

The speed of this perfectly correlates with the memory bandwidth of an M2 vs M5 Pro.

cagz•12 minutes ago

Is the 1/10+ reduction in memory usage applicable to larger models, i.e. would be possible to run a 200Gb model, by using 20Gb of memory?

memre12•8 minutes ago

Impressive if the numbers hold. Would love a table with per-token bytes read, measured SSD bandwidth, and cache hit rate—those four numbers would make the tok/s claims land better.

minraws•31 minutes ago

I have been working on doing the same for ling-3.0 seems very usable on my 5070 Ti now since it's only 5.1B active, you can even get pretty greedy and keep around 6% of each expert in memory and load the prompt and make the changes.

tracker1•about 1 hour ago

This is actually very similar to some ideas I've been having for a while... that having a smaller entry model that knows enough about "expert" models that themselves are smaller to hand work over to could be better/faster/lighter in terms of working through real problems vs the megalith ones we currently use. Highly distilled experts and coordination with a fallback mode to a larger model option.

docheinestages•35 minutes ago

Is there a pipeline or approach to do this to any model? I'm particularly interested in Qwen 3.6 27B as it's the best for its size at the moment.

rcarmo•about 1 hour ago

Would be awesome if it ran Qwen (the MoE probably won't squeeze that low, but...). This because I have hardly been able to use Gemma for any sort of useful coding.

dofm•14 minutes ago

Gemma 4's tool calling was recently fixed; that was the main issue with agentic use in my experience.

Otherwise IMO it codes about as well as the Qwen MoE for PHP and SQL. It's a fully impressive model (though it is not as mindbendingly impressive as the 12B, which is outrageously good for its footprint)

gitpusher42•41 minutes ago

You’re right, Gemma isn’t the best model for coding (afaik more "everyday tasks" related). My first idea was to use Qwen, but its architecture was much more complex to implement in this stack. I chose Gemma so I wouldn’t spend all my time debugging custom kernels and could actually move the project forward with simpler approach

jwr•39 minutes ago

In defense of this model, Gemma is actually a very good general-purpose model that can work with multiple languages. I use it for spam classification and for processing dictation, which means that I hold the entire model in memory all of the time, which is somewhat problematic (64GB RAM total, but heavy usage by docker, databases, etc)

trollbridge•21 minutes ago

Gemma is a great reference model and it’s easy to work with. Once you have Gemma working well, then do the extra work to use Qwen as well.

I am using Gemma for a few tasks simply because it’s “good enough”.

hsienchuc•about 2 hours ago

I've run local video generation models on an 8GB graphics card and know firsthand that nothing runs smoothly when memory is insufficient. So seeing 14GB of weights crammed into 2GB of RAM is impressive.

If running continuously for over an hour (like an overnight batch task), will a fanless MacBook Air overheat and throttle? Can the SSD handle the continuous weight reads and sustained output speeds?

Great work, congratulations on the release!

m00x•about 1 hour ago

This is where MoEs shine though. You don't need all experts in memory at once. Diffusion inference doesn't have sparse inference.

Pragmata•23 minutes ago

Would it be possible to use this for kimi k3?

what are the limitations

maxignol•about 1 hour ago

I'm really excited about what's been happening couple last weeks for local inference. I feel like it all started after colibri [1] was released. Great work !

Anyone got recommendation about what local model to use for what purpose ? I feel like (as they were saying in moonshot blog post [2]) each llm can be an expert in its own categories and with several small local we might get good coverage for decent usage, granted each one is specialized enough.

[1] : https://github.com/JustVugg/colibri [2] : https://fireworks.ai/blog/kimik3-fable

gitpusher42•about 1 hour ago

I think I first saw Flash-MoE (https://github.com/danveloper/flash-moe) in April. Huge respect to them, it was a big inspiration for this project!

huangsemao•about 1 hour ago

What part of the optimization process gave you the biggest speed gain?

mxmlnkn•about 2 hours ago

This sounds really cool. My intuition was that the selected experts might change heavily for each token, resulting in slow SSD loads for each token. This seems to be wrong. Did you create some statistics on how often the experts need to be changed? What is the longest token run without any expert change? What does such a token run look like? In which cases do experts change frequently?

gitpusher42•about 1 hour ago

The full route changes almost every token. The cache works through partial reuse, about 40% of experts repeat on the next token and 57% within two tokens, cutting I/O from 166 to 88 ms/token on M2 Mac.

The longest exact repeat we found was only two tokens. Coding tasks may have higher reuse if code related experts are selected repeatedly

cyanregiment•39 minutes ago

You’re a mad man - thank you!

Do I understand correctly that Ollama doesnt do that, and that’s why responses hang forever on a M3 running the same model through Ollama?

trollbridge•16 minutes ago

Please don’t use Ollama.

https://sleepingrobots.com/dreams/stop-using-ollama/

yakupov_bulat•about 2 hours ago

Wow, amazing!

What if there is enough RAM to fully load the model? I assume in that case I shouldn’t use your engine.

gitpusher42•about 2 hours ago

It depends on the use case.

I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s.

The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly comparable to ChatGPT’s speed for basic responses) my engine may be a good fit.

If you need maximum speed and flexibility just use MLX

0gs•about 2 hours ago

you could use mine ... github.com/0gsd/enough (it has other stuff too)

greggh•about 1 hour ago

It does exactly what it says it does. On my Mac mini M4 with 16GB of ram it is running at just over 5 tok/s. That jump from M4 to M5 is crazy.

gitpusher42•about 1 hour ago

What exact specs do you have? It might be because it's the 256 GB version. afaik, those versions have much slower memory bandwidth than the 512 GB models

My friend tried it on an M4 MacBook Pro and got 25–27 tok/s

greggh•2 minutes ago

Yeah, its the 256gb version.

giobox•about 1 hour ago

This is correct, the 256gb is substantially slower as uses fewer physical memory chips - less ability to read/write in parallel. The 512gb or larger models have substantially higher read/write rates, and typically performs 50-100 percent faster in benchmarks than the 256.

Was a primary factor in me buying a 512gb M4 Mac Mini, even though I planned to use large external SSD - I wanted faster spec boot volume.

h2aichat•about 2 hours ago

Hope you can do it for Windows users also (and small graphics cards). Thanks

gitpusher42•about 2 hours ago

Uh, I’m afraid it is Apple only. It is written using Apple’s GPU language, Metal, and heavily relies on the Apples’s shared memory architecture

Windows PCs would require a completely different approach

weras•32 minutes ago

Rushing to try it!

WithinReason•about 2 hours ago

Nice job implementing expert caching!

gitpusher42•about 2 hours ago

Thank you! Under good conditions it achieves approx a 67% cache hit rate with 16 expert slots

touwer•about 2 hours ago

Cool! Is there any info on this doing harm to the SSD? (Or other parts?)

wtallis•about 2 hours ago

Reads don't wear out flash memory to any meaningful extent.

gitpusher42•about 1 hour ago

AFAIK it should not because it is only reading

sscarduzio•about 2 hours ago

How does this compare to DwarfStar4?

fghorow•42 minutes ago

I'm curious too!

One obvious thing is that the memory requirements for this are substantially smaller than DwarfStar-- which AFAIK can only start to be used at 64GB ram and upwards. Another obvious thing is that antirez is pretty obsessed with making sure that DwarfStar passes all of DeepSeek V4 Flash's generating tests (loosely). I suspect that is also true of DwarfStar's implementation of GLM5.2, but I don't use that.

liuliu•21 minutes ago

DS4 is designed to do real-work. Gemma 4 is not going to cut it.

znpy•about 2 hours ago

I wonder if i can run this on my MacBook Neo!

gitpusher42•about 2 hours ago

I haven't tried it but it should work! You can try it and share your results, it would be really appreciated

I tried it on my wife's M1 MacBook Air 512GB and it gets 4–5 tok/s

Also, it must be easy to adjust for iPhones and iPads in theory

trollbridge•14 minutes ago

iPhones and iPads have much slower flash.

trollbridge•15 minutes ago

Yes, a Neo is roughly equivalent to an M1.