Nvidia Nemotron 3.5 Lightning and NeMo Switchyard

With the benefit of LLMs already being proven, in a couple of years we will have vastly better hardware for inference I guess.

I feel like now hardware is stagnating a bit, because the software side has moved too fast for the hardware to catch up. Once we settle on sone good, optimal software architecture for the models, dedicated hardware will easily increase throughout by 10x or 100x, for a fraction of the most.

LLMs seems quite simple, maybe we'll be able to print at home our own chips with the models.

I'm looking forward to seeing what types of new things people create over the coming years once there is less obsession with massive unwieldy LLMs. I think the incentives are just too strong to ignore.

This is 100% true for pretrains, likely true for RL as well although maybe there is some benefit to smaller activated params there. There is of course 0 benefit to small dense models relative to large sparse ones that are equally as memory efficient if you have enough computers.

Many on HN are in deep denial about this imo.

But yeah I'm skeptical all this overhead is worth it.

I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.

Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.

We can’t have legitimate debate if we have to assume a few bad actors are cloning their voice by the thousands, poisoning debate.

If we can pin one account to a real person, we won’t get rid of LLM-content and misinformation, but at least we can hold them accountable.

Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp).

Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs).

but

Qwen 3.8 27B is dropping this week...