Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

_davide_•2 minutes ago

I'm writing my own inference engine for Strix Halo and the same model. I already have 30%+ performance plus a more graceful decay over long contexts; that said, their point stands: memory bandwidth is what you really want.

amelius•2 minutes ago

Do unified memory CPUs suffer from the same memory shortages as normal memory?

I guess they're just welding the memory to the CPU chip, but still curious.

Havoc•about 1 hour ago

Think future generations of AMD could get quite interesting. They’re no doubt seeing people whining about mem throughput specifically

throwa356262•about 1 hour ago

"Can't" is not really correct.

Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.

lowbloodsugar•13 minutes ago

The current “big GPU” has 96gb of memory, but that’s not a “consumer GPU” apparently, while a $5000 Spark is a “consumer PC” I guess. In any case you’re probably better off running a large open weights model on the cloud.

LoganDark•about 2 hours ago

Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).

mountainriver•about 1 hour ago

Yeah this is such a funny thing going around. Try to run or train a small/medium sized model on a MacBook. It doesn’t go very good compared to a dedicated gpu

This is likely the right path in the future but it isn’t there yet today

bahmboo•about 1 hour ago

To be fair it's "only" half the throughput of a 4090 and a third of an RTX 6000. Significant but not an order of magnitude.

entrope•15 minutes ago

Those are the ratios for memory bandwidth, but the GPUs have a much higher ratio for compute, and that affects prefill rate / TTFT, right?

lowbloodsugar•18 minutes ago

An old ada Rtx 6000 maybe. A Blackwell RTX Pro 6000 is an order of magnitude faster and has 96gb.

ReptileMan•about 1 hour ago

>The two numbers that decide everything

Do you have LLM paranoia? The moment you read something in this style suddenly your brain starts doubting whether you are consuming human generated content.

cheevly•41 minutes ago

Maybe like… focus on the actual content instead of perceived writing patterns? Crazy I know.

glimshe•39 minutes ago

This. I don't particularly like the LLM writing style, but we've read a ton of very poorly written texts over the year with no complaints. LLMs are average writers with an annoying style, but not bad writers. If the content is good, I don't care if a LLM wrote it.

jazzyjackson•36 minutes ago

It’s a flag that no care went into it. Web surfing involves lots of little decisions following cues of “is this worth my time”

lukeinator42•36 minutes ago

The writing style is this staccato LLM-like style that is difficult to read as it has zero flow and meaningless sentences.

Like what does the second sentence even mean? Is it even a sentence? "The roofline math, the prompt-processing catch, the NPU red herring, and the owner-measured speeds."

forsalebypwner•6 minutes ago

Also so sick of the "it's not x, it's y", like stop fucking telling me what it's not! just get to the point!

` The mini PC's slowness is not a driver problem or a weak chip. It is arithmetic on the bandwidth number. `

abtinf•12 minutes ago

There is no doubt. It's junk.

OutOfHere•about 1 hour ago

Let's also ensure the SSD doesn't age prematurely.

craftkiller•about 1 hour ago

I was under the impression that when you're streaming the weights from disk because the full model won't fit in memory, that it is solely reading from the SSD, not writing, so it wouldn't be causing wear on your SSD.

kmeisthax•3 minutes ago

NAND[0] has a fun thing called "read disturbance" where repeated reads from disk will eventually flip 0s to 1s. You have to erase and rewrite the block before the bits flip[1], or you lose the data, but doing so is the same amount of wear as a write.

[0] I heard this being an issue with TLC, I don't know if it also applied to MLC or SLC.

[1] I suspect in practice they use an error correction code and rewrite blocks that read with corrected errors.

giantrobot•25 minutes ago

It is and it doesn't. You only get into disk writes if the system starts paging out to disk.

Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

⚡ Community Insights

Discussion (22 Comments)Read Original on HackerNews