Microgpt in pure C hits 10M tps on Apple m5

ddhorthy 1 day ago 27 commentsRead Article on github.com

ZH version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

87% Positive

Analyzed from 795 words in the discussion.

Discussion (27 Comments)Read Original on HackerNews

Retr0id•about 2 hours ago

> The most atomic way to train and inference a GPT in pure, dependency-free C.

What sense of the word "atomic" is meant here?

elromulous•about 2 hours ago

No dependencies, self-contained

MycroftJones•about 2 hours ago

Check out this port of microgpt to C, posted 5 months ago. It got a 2500x speedup over the python version. https://github.com/moebiusV/cugpt

gok•29 minutes ago

It could probably go quite a bit faster using Arm SME. The entire network could fit into the ZA register.

ilaksh•about 3 hours ago

This is not an LLM obviously , it's just for generating random names. But interesting to think of the possibilities of truly tiny language models if there were connected together.

alightsoul•about 2 hours ago

It's an slm (small language model) due to number of parameters and it uses the same architecture as an llm, but llms have billions of parameters

nomel•about 2 hours ago

Hasn't it been repeatedly shown that many small models perform worse than a large model of the same total parameters?

reilly3000•27 minutes ago

I think that is a sort of a reverse scaling fallacy. Given the right resources and environments, many small models can function together in an emergent way. I’ve been on the lookout for an SLM version of Conway’s game of life. SLM always reminds me of slime molds, which demonstrate a form of intelligence which is remarkable.

dcow•about 3 hours ago

Is token rate a function of parameter size?

unrahul•about 2 hours ago

Yes, a quick back of the envelope math is 0.65 * (memory bandwidth of the card / (model weights in bytes + kv cache in bytes) ~ practical decode tps. Below context around 32k (depends upon the model but again can be used as a placeholder number) you can ignore the kv cache in bytes and the math becomes just about memory bandwidth and model weights in bytes.

rbanffy•about 2 hours ago

Not quite linear, but yes.

api•about 3 hours ago

Isn't a MoE model basically a cascading tree of smaller models or some variation of that?

unrahul•about 2 hours ago

You could think of it as a standard decoder only LLM (almost all modern ones we use everyday), with some layers (experts) having parallel networks and conditionally based on the input token (per token) - the token is routed through some of these layers. In the case of a non MoE (dense) - each token goes through all layers, so the inference engine has to read all the layers and do a matrix (layer) times vector (token) computation, while in the case of MoE the number of layers per token that has to do the compute is substantially lesser, so one can expect much higher tps than a dense model at the same number of parameters (size - 7B, 27B etc)

pkilgore•about 2 hours ago

Honestly not sure this is impressive. I ported microgpt to zig as a learning exercise, then moved scalar engines to NEON/metal just to see what happened. Besides metal being slower (I probably did something wrong, but it could be due to the fixed costs of memory transfer into the GPU not being worth it due to the small model).

Anyways, it was also stupid fast, particularly compared to the python version. But I was pretty sure that's irrelevant to real production architectures!

throwa356262•about 4 hours ago

And the 5 years old AMD Ryzen 5 5600H is doing 7M?

Am I reading this right? Then I need to try this on Strix Halo

rbanffy•about 2 hours ago

And it's only using AVX-2 and not AVX-512, AMX or ACE. Or built-in GPUs and NPUs (the M series doesn't emphasize matrix multiplication on the CPU side because it already has matrix multiplication units on the GPU, which is always attached).

ranger_danger•34 minutes ago

But do processors actually offload any CPU opcodes to their GPU? That could be quite useful if it can be used to improve execution speed.

bigyabai•about 2 hours ago

Before the M5, there was no dedicated matrix multiplication hardware on the Apple Silicon GPU. Their solution was generally using the NPU and AMX coprocessors for tensor and matrix workloads.

fwip•about 3 hours ago

Model is 4K parameters - I don't know enough about that size of model to know if this impressive or not.

altcognito•about 3 hours ago

It's a trivial example. This won't be useful outside of a VERY specific domain without more parameters. Many people need to know about the bitter lesson.

https://en.wikipedia.org/wiki/Bitter_lesson

Over time, I'm sure we'll be able to filter information better and get parameter counts down, but I wouldn't count on that within the next 6 months.

odo1242•about 3 hours ago

The point here is that the library's overhead cost is very low. The fact that a tiny model can reach 10M tokens per second means that the overhead of token decode, memory allocation, calling the model, etc. is very low. The model doesn't actually need to be useful to prove that point.

entrope•about 1 hour ago

Is it that impressive? It is a model generating short strings from scratch, so I do not think there is significant token parsing or memory allocation going on.

In compute performance, 10M tok/s * 4096 parameter/tok * 4 byte/param = ~160 GiB/sec implied memory bandwidth, vs nominal ~300 GB/sec for the M5 Pro's RAM. That seems even less notable once we consider that 4096 parameter * 4 byte/param = 16 KiB of parameters fit easily within L1 data cache (the M5's efficiency cores each have 64 KiB of L1 data cache). 40 GFLOPs means ~10 FLOP/cycle, about 2.5 NEON instructions per cycle given that the core instructions are fused multiply-adds (FMAs). A random web page I found says the M5 family cores have a 4-wide SIMD block, meaning this gets about 63% utilization: respectable but not super high.

voakbasda•about 2 hours ago

It’s interesting and worthy of genuine applaud for being a good starting point for further work.

That said, I am more interested in what size model this could manage while producing “just enough” tokens per second to work at a conversational rate. What are models in that class capable of doing for me?

altcognito•about 2 hours ago

Thank you for your kind reply. I appreciate your point completely and while I tried to moderate sounding dismissive of what was being done here, I think I could have done better.

I love "trivial" examples and everything you've said is tue.

alightsoul•about 2 hours ago

I think the bitter lesson only talks about task performance but not computational efficiency. Could tiny models improve efficiency? Maybe by just using a general architecture on specialized data, so the artichecture itself is not task specific?

mcbuilder•about 1 hour ago

Small # of parameters means no memory bottleneck, which means blazing fast performance.

andai•about 1 hour ago

Wait, does this fit in L1?