Unsloth Dynamic 3.0 GGUFs

The reason for running those insanely low quants is to fit in extremely limited memory budgets. The first thing you sacrifice is speed, then context and accuracy (up to you in which order). IQ2_XXS and below is desperate/proof of concept territory. If you have a spare half gig for the MTP drafter, run a larger quant instead, it will be less incoherent, and damn the speed, it won't be garbage at least. Only around Q4 I'd allocate the comparative luxury of more memory for a speed increase. At least on a dense model. MTP makes a lot more sense (but helps statistically a bit less) on an MoE.

Qwaiting for that 3.8-35B-A3B

Hey we did not remove the MTP for sizes above 8GiB - but yes for small GGUFs under 8 ish GiB, we removed the MTP module (IQ2_XXS and lower), because it's 500MiB to 750MiB in size, and on small 8 GiB machines, even 500MiB is needed.

As someone in the comments said we made a separate Q4_0 MTP if that's helpful so you can use that.

But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL

Q2 quantization is basically giving a capable model a lobotomy. It will not accurately represent how smart or capable something like qwen 3.8 27B in Q8 will be.

you can still have it, no?

> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed

We made something called Divergence-300 @32 (and later @512) which tests actual inference across 32 tokens on a held out test (Terminal Bench, DeepSWE, Math etc)

We do plan to do larger benchmark suites though!

I tested Qwen 3.8 on the Blade CTF last night, it took 3 hours but got the correct answer.

I know that didn’t answer your question but I was looking for a test suite and couldn’t find anything.

After reading the logs, there is far less doom looping than with 3.6, but whether that’s a one off or not is up for debate.

Q4_K_P

I tried some 1-bit, 2-bit, and bonsai quants against closed eval sets. They were essentially useless for my case. The little errors accumulate and send the whole output off track quickly.

If you had some use case with very small output sequences they could be interesting to try. I think dropping down to a 9B-class model would produce better results for most cases.

Not 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok/s average for the first 32k of context) that the speed hit might make it not worth the extra smarts

Unsloth use a property dataset they don't release, however you can indeed create quantisation locally on your machine and it's pretty easy, llama.cpp comes with everything you need.

Just quantizing takes seconds-to-minutes, llama.cpp provides a nice tool[0]. Improving quality is then a matter of picking specific tensors to maintain at higher accuracy, checking on representative data, and repeating.

[0]: https://github.com/ggml-org/llama.cpp/blob/master/tools/quan...

Quantization is typically very cheap and fast. It can even be done on hardware that does not fit the model, by processing the weights layer by layer.

I use this project: https://github.com/vllm-project/llm-compressor

What size context are you able to squeeze in with less than 2gb of headroom? I have had some luck using a quantized kv cache but i fear that also decreases overall quality.

Yes, google "llama-server split model multiple GPU" for some concrete examples.

Yes, and if you have the PCIe lanes (say, an x16 lane - actually delivering 16 lanes! - to each GPU) it's also quite performant - it's called a tensor split in llama-server.

If your motherboard/cpu doesn't actually have those (few do outside some xeons, epycs and threadrippers) you can still do it - it's called a layer split and will work even with 1 lane per GPU. Each GPU will work at its maximum speed, but only 1 will be active at any given instant - imagine a relay race.

(Didn't mention which PCIe generation - obviously the higher the better. At v4 and up, even 8 lanes per GPU would be enough for a performant tensor 4-way split)

Edit: If you have more than 1 user at a time, the GPU can actually all be working all the time, if there are enough parallel requests to serve. But you need enough KV cache for all the sessions you're running in parallel.

Yes, you can. Ideally though, you want to minimize the number of cards and maximize the amount of memory in each card.

Using multiple cards is one of the things that the models and software that Unsloth releases does really well in terms of ease of use and relatively good performance.

I do use 2 amd gpus and I get high 40 for generation, 500 for pp and low 20/100 by the end of the context of 256k.

llama-server --host 0.0.0.0 --port 8089 -m Qwen3.8-27B-UD-Q8_u.gguf --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-n-min 1

if you have an igpu and want to exclude or just use some gpus you can use

--device Vulkan3,Vulkan2,Vulkan1

in my case vulkan because of amd, you can see your devices with

llama-server2 --list-devices

Available devices: Vulkan0: AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) (33515 MiB, 29349 MiB free) Vulkan1: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 4911 MiB free) Vulkan2: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 7681 MiB free)

I am getting 14 t/s on my 16 GB card at full context with the UD-Q3_K_XL quant. Model link: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF.

Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.

Yes, but unless they support NVLink (they don't), it's quite slow.

What about do you mean by single threaded? Each token is predicted by using parallel computation on the GPU.

That's not how that works. Selecting a different token is not inherently erroneous. A correct solution can still be found despite divergence.

I don't think you can extrapolate that measurement across multiple sequential draws like that. We presumably are comparing against a single trajectory rather than a tree of trajectories. So once we make the wrong choice and step off of the blessed path, we have no way to assign a ranking to the next token; it's error is undefined.

I've seen LLMs self correct in chains of thought ("because of foo and bar, I need to... Wait, bar is not true, so that won't work") so I have to imagine this is a massive overestimate, errors do not necessarily compound.