Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint

122

JJonSchneider about 3 hours ago 37 commentsRead Article on prismml.com

DE version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

91% Positive

Analyzed from 1165 words in the discussion.

Discussion (37 Comments)Read Original on HackerNews

jedbrooke•26 minutes ago

Running at about 7-8 tok/s (~60 tok/s prefill) on a Mac Mini M2 16GB.

So far feels smarter than Bonsai 1 27B, it’s slightly larger than the Q1_0 quant. Super exciting stuff :)

simonw•about 2 hours ago

If you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's llama.cpp fork to get them to work, from https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-...

This should work:

  cd /tmp

  # Get the Prism macOS runtime
  curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz
  tar -xzf bonsai-runtime.tar.gz

  # Get the ~5.95 GB GGUF model:
  curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf

  # Run the server, I used port 8331
  ./llama-prism-b10685-7dffb15/llama-server \
    -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \
    --port 8331 -ngl 99 -fa on -c 32768

Then open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this:

  uvx llm openai endpoint http://127.0.0.1:8331/v1 \
    --model bonsai-2-27b --responses hi

That's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said "ggml_metal_device_init: - the tensor API is not supported in this environment - disabling".

simonw•about 1 hour ago

I used that to Generate an SVG of a pelican riding a bicycle:

https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

It took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.

kadoban•about 1 hour ago

Honestly looks pretty good except whatever is going on with its booty. Is that an ass helmet? I cannot parse what's going on there.

Forgeties79•about 1 hour ago

I think it’s supposed to be a wing

refibrillator•about 1 hour ago

Where did you get these instructions?

They have a demo repo with a setup.sh script:

https://github.com/PrismML-Eng/Bonsai-demo

The release tag and weight file you suggest doesn’t match what they wrote.

simonw•about 1 hour ago

I figured them out, starting from the GGUF on Hugging Face.

If you have found better instructions and they work then use those instead!

Personally I prefer to download models directly rather than running some `./setup.sh` script where I need to then review what it does first.

refibrillator•34 minutes ago

Yeah just wanted to mention in case it explains the 2x lower throughout you are seeing on M5. To be fair their documentation is a bit inconsistent in some spots.

Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.

nikwen•17 minutes ago

It would be great to have upstream llama.cpp support for this!

OutOfHere•about 1 hour ago

Could you also note all this in a Discussion message at https://github.com/PrismML-Eng/llama.cpp/discussions ? Or are you only a HN person?

adrian17•about 1 hour ago

> Ternary Bonsai 2 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 effective bits per weight

If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what's the special sauce that makes them better?

https://news.ycombinator.com/item?id=49611128

edflsafoiewq•about 1 hour ago

I think the general idea is naive quantization falls apart below 4bpw but you can go lower with more sophisticated QAT-adjacent methods. Bonsai's quantization method is proprietary though.

danbrooks•about 1 hour ago

Nice! Does anyone know how this compares to the Unsloth quantizations of this model? https://unsloth.ai/docs/models/qwen3.8#run-qwen3.8-guide

Havoc•32 minutes ago

Cautiously optimistic. The V1 was noticeably weak on world knowledge but here the 3.8 base model is geared more towards reasoning than world knowledge anyway so might not matter as much

miffy900•about 2 hours ago

I really wish people would stop saying N times smaller than something when making a comparison; that makes no sense - it's 1/9th (11.11%) the size. You don't get a smaller quantity by multiplying by a number greater than 1.0. You could instead reverse the subjects being compared - "the original model is 9x bigger than this new smaller, efficient model" or some such. That makes sense.

I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.

zamadatix•11 minutes ago

I agree it makes little sense in a literal mathematical take but "it's 9x smaller" is too much of linguistic advantage compared to "the original is 9x larger" to expect a change with. You don't have to invoke fractions, it keeps the thing in focus as the first subject, it matches the pattern of the inverse statement, and it's just plain shorter... so that's what people will do and adapt the meaning to be.

One other way to map both types of linguistic statement consistently to math is to interpret "9x" as "there is a 9 times difference between these two things" and then smaller/larger tells you which end of that separation the subject is rather than specifying whether the multiplication builds up or down.

hamandcheese•about 1 hour ago

If we were talking about speed instead of size, i think it would be perfectly reasonable to say 9x faster. I'm not sure I agree that 9x smaller is unintuitive. It makes sense to me.

_carbyau_•23 minutes ago

"faster" relates to speed. Speed is related to time and speed of a thing is usually defined by time. 9x faster speed translates to time/9. There is an extra step of related conversion there.

Conversely, 9x [filesize/natural number] is bigger. Every time. At least in the basic maths used by most people. There is no conversion into other units.

Therefore "9x smaller" when talking about a natural number like filesize is a nonsense statement in logic terms. If you strive for unambiguous phrasing - which is a significant part of the programming experience - this logical nonsense might well perturb you.

But english language is a flexible thing and if the phrase communicates your intent to your audience then that's fine by me.

simondotau•23 minutes ago

"Nine times" literally means multiplied by nine, but here we're dividing by nine. It's not unintelligible (because the corrupted verbiage is so commonplace) but it is needlessly awkward. Like saying "resulted in a size reduction increase of 10 megabytes."

UI_at_80x24•about 1 hour ago

Me too!! It's a huge pet peeve. And it's so hard to get people to see how it's linguistically AND mathematically WRONG.

Aurornis•about 2 hours ago

These are small enough that you can run them entirely in the browser https://huggingface.co/spaces/webml-community/ternary-bonsai...

Remember to clear the downloaded weights afterward.

Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.

outofpaper•about 2 hours ago

So you have some fun examples?

JonSchneider•about 2 hours ago

I'm hoping they release an 8B v2 based on the Qwen 3.8 series in the near future - that would give us a really powerful model that could be run directly on users phones.

verdverm•34 minutes ago

That would require Alibaba releasing a Qwen 3.8 8B first

sroussey•about 1 hour ago

Yes! And maybe get a hf fused webgpu runner for that model so it’s fast!

kamranjon•about 3 hours ago

Love this for the folks with 16gb graphics cards - 3.8 27b has been incredible but not quite runnable on anything less than 32gb - will try loading this up on my 16gb intel b50 and see how it goes - not sure these quants can be accelerated by the XPU cores yet but maybe in time!

kadoban•about 2 hours ago

You can run the ~4 bit quant(s) on 24gb, if you're not _too_ picky on context size.

This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.

spijdar•about 2 hours ago

I run Unsloth's UD-Q4_K_S on 20 GB of VRAM (RX 7900 XT) and I get ~90k tokens of context without quantizing KV cache. With 8-bit quantization, I get about a 134k token context window. That's with only one slot, but for me, it works pretty darn well, with 20-35 tok/s depending on how full that window is.

logicallee•about 1 hour ago

(In case anyone remembers the compression post from yesterday[1], I checked and this one doesn't qualify for further compression - it's not zero-biased at all.)

[1] https://news.ycombinator.com/item?id=49732931

abraxas•about 2 hours ago

I'm not following the local mdoel scene too closely but this seems quite amazing. Is this able to be run on Apple silicon too?

Havoc•about 2 hours ago

Their first 27B bonsai was able to run on an iphone.

kamranjon•about 2 hours ago

"Ternary Bonsai 2 27B reaches up to 143 tokens/second on NVIDIA GeForce RTX 5090 and 46.8 tokens/second on M5 Max. On an RTX 4090, Ternary Bonsai 2 27B consumes just 0.714 mWh/token, making it 40% more energy-efficient than an 8B model running in full-precision."

pizza234•about 2 hours ago

Their mention of the 5090 is bit odd, since on 32 GB GPUs, Q6 fits while having better quality. Very interesting model for 16 GB GPUs though!

sisve•about 2 hours ago

They mention 5090 with regards to speed, Q6 will not have that speed?

And speed matters a lot for many use cases

azatom•about 1 hour ago

it is like "my fridge is 2mkm (millikilometer) from my desk" m=0.001 h=3600 it should be just Ws or just J

_kulang•about 1 hour ago

What’s wrong with milliwatt hours?

z2•about 2 hours ago

I'd love to see a Bonsai model start with a 100B+ parameter model and get that down to <30 GB. But maybe at that point we call it Topiary?

all2•29 minutes ago

Other names that occur to me: Orchard, Forest, Stand (of trees).

2001zhaozhao•about 1 hour ago

I think if they made this for Qwen3.8-Next it could fit in a single 5090?