AMD acquires Taalas to boost inference performance by etching models in silicon

iitvision about 2 hours ago 24 commentsRead Article on theregister.com

DE version is available. Content is displayed in original English for accuracy.

https://ir.amd.com/news-events/press-releases/detail/1296/am...

⚡ Community Insights

Discussion Sentiment

93% Positive

Analyzed from 1572 words in the discussion.

Discussion (24 Comments)Read Original on HackerNews

bob1029•about 1 hour ago

I feel like NAND process tech could become useful at solving some of these problems. A GPU where you can update the weights a few thousand times may be sufficient.

mdp2021•36 minutes ago

The basis of Taalas is "compute in memory" electronics - past Von Neumann's separation of processor and memory.

You need to be able to add|mul where the data (the weights) are stored.

addaon•44 minutes ago

NAND hasn't been scaling great lately. It seems like PCM or MRAM would both be better fits.

kridsdale1•about 1 hour ago

FPGA model storage?

mikeayles•about 1 hour ago

AMD could have saved their money and used their own hardware! I've got a language model doing 60k tok/s on AMD hardware already, a Xilinx Kria K26 SOM, with the weights baked into URAM/BRAM with zero DRAM in the token loop. Same thesis as Taalas: single-stream decode is bandwidth bound, so stop fetching weights from far away.

Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though.

I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two

tandr•about 1 hour ago

Well, technically it is their hardware now...

questionableans•34 minutes ago

And their team, if they treat them well.

A_D_E_P_T•about 1 hour ago

This is probably a win-win. The team gets paid, and we get greater assurance that their best ideas and architectures -- which are truly impressive -- are going to see the light of day in actual products.

badatnames•about 1 hour ago

They were too small for this to be a meaningfully sized purchase for AMD, there's real risk they get sucked into a team that ultimately delivers sqat, not to mention the chances of anything being delivered in an even remotely consumer-priced bracket are definitely out the window

whythismatters•about 2 hours ago

The demo: https://chatjimmy.ai/

walrus01•38 minutes ago

I know it's a relatively tiny model, but damn, is that thing fast.

It also mostly passes the "schlong" test

https://pastes.io/YcxSi8Fp

AussieWog93•15 minutes ago

I read the paste, it got the etymology wrong, no? Schlong comes from shlang (snake), not shlemp (is this even a word? I don't speak Yiddish but couldn't find it on Google).

Oxford also claim that its first recorded use was from the 60s, not the 20s; https://www.oed.com/dictionary/schlong_n?tl=true

thoughtpeddler•33 minutes ago

I didn't realize there was a SchlongBench™ (but of course there is). What's it test? (asking seriously)

walrus01•30 minutes ago

There isn't SchlongBench(TM) yet, it's a specific question I've been asking of differently sized models as a randomly chosen gauge of how much less commonly used knowledge is perma-baked into it. In this case a question about a specific yiddish origin slang term. Small/bad models don't know it's from middle high german or Yiddish and get its origin and meaning totally wrong (or it runs into model censorship related to slang related to the male anatomy).

It's also a question I have found will cause models that don't know what it is to go off quickly in a direction of hallucination trying to explain it, so the hallucination is evident very quickly starting from the first ever prompt issued with 0 context fill. Example: I had a model write four detailed supposedly-accurate sounding, grammatically correct paragraphs saying its origin is from AAVE (African American Vernacular English), which it most certainly is not

You could do the same by picking any topic that is very rarely discussed in conversation, some esoteric and narrow piece of knowledge and asking the model about it.

wxw•about 1 hour ago

I freakin' love this demo. It feels magical.

VBprogrammer•33 minutes ago

I had the same reaction but then I showed it to my partner. She completely didn't get it, in her words "how can it be thinking of a good answer when it's that quick?"

I tried to explain but I fear were probably going to be adding artificial sleeps to these things to convince the masses it's doing something clever.

axus•5 minutes ago

I asked it some old hardware command line questions I'd recently asked Gemini, it hallucinated parts of the answer.

The characters in the 3-act Shakespearean play had very little depth, many of the names were similar, and they were not very smart, but the simple plot was cohesive.

varun_ch•11 minutes ago

to be fair, the model used for Chat Jimmy is not very smart, but the world where it is smart is very interesting.

It’s going to be really crazy when the bottle neck for agents is the speed of the tool calls rather than the speed of inference. Imagine an agent interacting with the terminal near instantly…

senderista•35 minutes ago

Wow, feels like Google web search in 1999.

joshvm•1 minute ago

If you still want the experience, go and browse McMaster Carr. Wizards designed that website.

hendurhance•37 minutes ago

I understand the appeal due to the speed

itvision•about 1 hour ago

OMFG this thing is fast.

phoh•18 minutes ago

its fast but try to get it to give you pi to 50 decimal places. it didnt go well for me.

walrus01•15 minutes ago

I think the same exact model running on CPU-only and RAM, or a small GPU, would do about the same? It's quite an old model now and small, you could throw a GGUF into llama-server or something for a side by side comparison.

https://huggingface.co/meta-llama/Llama-3.1-8B

As I remember just about any english language model from mid 2024 and earlier didn't even do well if you asked it to count sequentially from 0 to 100, nevermind calculating stuff.

nsxwolf•about 1 hour ago

It doesn’t believe it’s running on that chip, it’s arguing with me

shaewest•about 1 hour ago

It's running a very small, non-reasoning model at the moment. But more generally, almost all LLMs argue on the hardware/model they are/are on.

metadat•40 minutes ago

What would tokens/sec performance look like for a reasoning model? An order of magnitude slower?

dumberquestions•about 1 hour ago

Which model? Or how many active parameters?

syntaxing•about 1 hour ago

Honestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.

encyclopedism•about 1 hour ago

Imagine a multi-modal model with 1000's of tokens per second. Realtime inference for a host of applications. This is a BIG deal and will change the landscape in unfathomable ways.

The https://chatjimmy.ai demo was impressive.

Once models settle down this makes sense. Imagine a cartridge with a physical model on it. You purchase a cartridge and stick it in your computer/phone/server. Want to upgrade? By a new 'cartridge'.

This should bring inference cost down dramatically, I wonder how OpenAI/Anthropic feel about that.

2001zhaozhao•25 minutes ago

i'm looking forward to Qwen3.8 27B launch to see how much models have peaked at a given size.

it might already be time to start burning the best small models onto hardware since it's possible they can't get much better at many tasks like knowledge recall due to the inherent information density limits for models at a given size.

anthonypasq•18 minutes ago

very interesting idea. i didnt think of that. i was just assuming youd have an additional one of these in your phone for actual lightning fast local inference

Grosvenor•34 minutes ago

> Imagine a cartridge with a physical model on it.

I can finally have my own Dixie flatline. Cool.

mdp2021•2 minutes ago

> Dixie Flatline

In case some did not know: also the movie (or TV series?) is finally happening.

# Neuromancer - Official Teaser https://news.ycombinator.com/item?id=49055037

breadislove•13 minutes ago

we have not converged at all, if you look at how different the chinese models in terms of architecture you can guess that the labs are experimenting a lot as well. we are seeing all different types of hybrid architectures, different attention methods and so on. Of course on a high level its still a transformer but if you take a proper look we are seeing more divergence then a convergence.

kevin_thibedeau•about 1 hour ago

Then we can have machine psychologists pull cards when they run anok.

VladVladikoff•about 1 hour ago

Wouldn't this mean someone with sufficient hardware could lift the SOTA model weights off the chip? Or are you saying that these chips would only be used internally by these companies and not sold to the public?

dumberquestions•about 1 hour ago

I wouldn't expect companies not sharing their weights today to be any more likely to share them if they're on hardware, this doesn't sufficiently hide weights from a local user.

snek_case•about 1 hour ago

The weights are very unlikely to be on the chip itself. That wouldn't work for SOTA models that are terabyte scale, even quantized. This is probably an accelerator for specific kernels in the model, but the weights are likely loaded from memory. The chip may have SRAM to store some of the weights temporarily during inference.

foltik•2 minutes ago

At least in the case of Taalas the weights are physically encoded directly on the chip.

It’s composed of 4-bit multiplier cells that compute all 16 possible results in parallel, and the top metal wiring layers physically route the single one corresponding to that cell’s weight to the next layer.

syntaxing•about 1 hour ago

I don’t get why this is an issue? You can run Claude/OpenAI SOTA models through Amazon bedrock. These weights have to live somewhere to run on Bedrock.

amazingamazing•about 1 hour ago

One idea would be to use an open model.

smokel•about 1 hour ago

The technical aspects of SOTA models are not publicly documented. How do you know if something is converging?

syntaxing•about 1 hour ago

SOTA American models are not. SOTA Chinese models are. From a physics aspect, closed source models cannot be too far from open source ones in terms of size. There’s only so much you can squeeze out a B100 style cluster even with fancy Dflash style diffusion model for the speculative model.

_aavaa_•about 1 hour ago

If we had deepseek v4 flash 0731 etched on a chip it would be more than capable enough and fast enough for so many people's needs, even hardcore engineer.

nurumaik•about 1 hour ago

Will be capable and fast enough for 2-3 weeks until new sota drops

cyanydeez•about 1 hour ago

if they were still exponentially increasing, they wouldn't be preparing for an IPO. IPO is where companies go to die and founders escape.

walrus01•34 minutes ago

Having a base model ASIC as a physical piece of hardware makes me think of the early days of microcomputer desktop stuff where having a socketed ROM or PROM was a key piece of hardware, and people actually knew/cared what ROM was on their system's motherboard.

Imagine if like instead of having a specific Mac Plus ROM, you had a thing that looks like a fat ASIC that can hold models sitting on a slotted daughtercard directly next to the CPU and RAM.

cyanydeez•about 1 hour ago

I don't think there'll be a fine tune card; you'll have the base model vintage whatever year, and then your GPU will do whatever LoRA layers you want it to do; the LoRA will wrangle older dated models into the current of whatever your looking at.

But yeah, for things like programming, if it can do linux and python and some go and sql and javascript, larger domains can be threaded with LORA

bhouston•about 1 hour ago

Toronto Canada startup btw.

kridsdale1•about 1 hour ago

Works well, I remember driving by the ATI building as a kid.

cmrdporcupine•about 1 hour ago

Seems to be somehow some kind of offshoot from or connected to Tenstorrent, which is just down the road. Founder looks like he was/is maybe at Tenstorrent and previously associated with Keller?

Always fantasize about applying at Tenstorrent, but wrong side of Toronto. 2 hour commute.

MarkWayneNewton•about 2 hours ago

While this design is self-limiting I think its a good approach. It doesn't take an entirely new architecture or infinite memory to produce significant performance improvement.

proxysna•about 2 hours ago

Really hoped to see their hw out in the wild one day

badatnames•about 2 hours ago

Well so much for that dream.

Guess we can look forward to picking these up ex-enterprise on ebay for under $5k a pop in a decade or two

ycui7•about 1 hour ago

so qwen3.x-27b on hardware? or better deepseek-v4-flash on hardware .

ilaksh•about 1 hour ago

I wrote them an email asking for PrismML Bonsai 27b Ternary which is like 6b or something crazy small and would be a lot easier for them to do initially.

mdp2021•22 minutes ago

They were specializing their forthcoming system on 4-bit FP - which I understand is a structural decision.

Bonsai Ternary (1.7bits/weight) is a compromise, compromise that has to make sense in the context - efficient when translated into transistors.

rvz•about 2 hours ago

Didn't even give them a chance to launch the hardware.