DE version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
75% Positive
Analyzed from 1400 words in the discussion.
Trending Topics
#model#cache#models#more#same#routing#ram#prompt#caching#https
Discussion (59 Comments)Read Original on HackerNews
I'm looking forward to seeing what types of new things people create over the coming years once there is less obsession with massive unwieldy LLMs. I think the incentives are just too strong to ignore.
I'm having an extremely hard time thinking of companies that have prospered due to optimization. Most of them were swept away by hardware advances, instead.
And I'd argue "hardware advances" are more proof of optimization.
I think when successful, optimization really just means 'finally built right' and people forget the ridiculously inefficient ways before.
While I do love optimized software, the hardware side, especially for PCs, has been stagnating for way too long. At least now we have a valid use case for doubling available RAM every 2-3 years again.
I had a reasonably beefy Lenovo consumer line laptop that I bought in 2011, 8GBs of RAM. Its screen hinge broke and I couldn't repair it but I'm fairly sure it was otherwise still usable in 2023-24, once the HDD was replaced with an SSD. I think even now entry level laptops are sold with 8GB of RAM.
By comparison a PC from 2000 was utterly unusable in 2012-13.
> When deployed, NeMo Switchyard can intelligently direct each request to the most capable and suitable model for the job
How do routers like this handle prompt caching when you send the second request?
Sticky models per session? but then the second message of that session won't be sent to a suitable model, and will only be sent to the same model as previous one.
I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.
Prompt caching isn’t about caching the literal text of the prompt. It’s about caching the result of running prefill on the prompt (or, equivalently, the result of generating the prompt one token at a time by autoregressive inference, or some combination of the above in the case of speculative decoding). This is often called the “KV” cache, and it is very model-specific.
But yeah I'm skeptical all this overhead is worth it.
Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.
prompt-cache won't work with these
You decide to switch to DeepSeek in the same session via the model picker, and continue as usual. What happens is that the cache for DeepSeek is created with the 200k + the incremental message. After this, cache can be kept warm for both models; two instances of the cache exists, one for GPT and one for DS.
You switch back to GPT. The whole session is sent to the model with the 200k original from GPT and the incremental messages you sent to DS. The 200k is read from cache and the incrementals are new, and then added to the cache.
Let's say every second message you switch between GPT and DS; cache was 200k and each incremental message is 1k. If you kept going with only GPT, cache hit rate would be 200k/(200k+1k) = 99.5%. When you switch between two models with warm cache, hit rate instead becomes 200k/(200k+2k) = 99%.
Model routers work the same way. Keep the cache warm, replicate it in two places. For this reason, when you set up your model pool for routing, you want to keep the model pool small and differentiated.
First principles of model routing: https://try.works/first-principles-of-model-routing
role-model router and protocol: https://github.com/try-works/role-model
note: edited to keep the answer to the below message clearer
- problem: massive deluge of information because of AI
- solution: human beings should adopt a minimalist style of communicating in writing.
- e.g. this entire website page can be ten bullet points.
We can’t have legitimate debate if we have to assume a few bad actors are cloning their voice by the thousands, poisoning debate.
If we can pin one account to a real person, we won’t get rid of LLM-content and misinformation, but at least we can hold them accountable.
Or do you have something else in mind ?
I'm not sure people really want that
https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvi...
but
Qwen 3.8 27B is dropping this week...
Even though that model is already "old", qwen was way ahead everyone else in that size category before this Meta model.
Also, probably for non-Chinese usage, using a non-Chinese model might lead to better results.
The suite is across many categories, not only coding, and most of the tasks are low-horizon (or what the opposite of long-horizon is), where the max thinking time is around 10 minutes.
Gemini models are really smart, unfortunately they don't play well with any harness, so hard to use in practice.
But try them out for one-shot tasks, they are really good. Don't use them for coding in a harness, but you can ask them to generate code/planning (still, for coding only other models are indeed recommended).
The cost is similar vram footprint I guess (?)