Mercury 2.5 LLM hits 770 tokens per second

Discussion (9 Comments)Read Original on HackerNews

bearjaws•about 1 hour ago

If you care about speed Cerebras gpt-oss-120b is 1400tk/s and "just as smart" in ranking.

I've used it on a few for fun projects and its decent but the speed is crazy to watch.

walrus01•about 1 hour ago

Pricing at $0.25 and $0.75 already puts its cost well above reasonably reputable inference providers for deepseek v4 flash or qwen 3.8-flash-next or similar class of open weight LLMs that fit in under 170GB of RAM, so I don't see the point. I think this is probably also stupider than laguna s 2.1 which can also be very cheap to serve.

nylonstrung•35 minutes ago

I honestly think the diffusion LLM approach is a dead end

It's telling that frontier labs like Google toyed around with it but didn't invest further even for their most speed and cost sensitive small models

Still unclear for what, if any use cases this is pareto frontier

LarsDu88•21 minutes ago

You can't think that a small startup versus Anthropic's training setup is anywhere near the same scale to make apples to apples comparisons.

Not sure how the Chinese labs pull it off though using autoregressive models. The secret sauce is probably going to be in the training data.

The main reason Google hasn't switched over to DiffusionGemma is because serving at larger batch sizes loses the speed gains you get from diffusion, and most of the primary use case is serving many users at once off a single device with a large batch size.

If you were to move to on-device low latency... like say in a robot or something, then the story might be different...

rvz•about 2 hours ago

The speed means absolutely nothing when it is finishing almost dead last when compared to the frontier AI companies.

copperx•about 1 hour ago

Ah, the old "good, fast, or cheap; pick two" proves true once again.

downrightmike•about 1 hour ago

Give it a few months.

voiceeh•30 minutes ago

Not if your use case needs speed. For one of my products I can't use an LLM that has a p99 of >700ms for TTFT.

glouwbug•about 1 hour ago

Some of us want fast food

Mercury 2.5 LLM hits 770 tokens per second

⚡ Community Insights

Discussion (9 Comments)Read Original on HackerNews