HI version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
90% Positive
Analyzed from 4677 words in the discussion.
Trending Topics
#models#model#small#more#better#tasks#enough#local#don#where
Discussion (122 Comments)Read Original on HackerNews
There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on.
Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do.
I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.
The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words.
The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Recent comment touching on this in relation to LLM's in more depth: https://news.ycombinator.com/item?id=49322695#49323341
Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LLMs could eventually outperform stockfish. It’s not true today and it remains to be seen whether it will ever be true. For now, specialized models are absolutely better at specialized tasks.
If you look at value as purely the LLM output, then there's a valid argument that the best frontier models will always be better than fine tuned specialists. (I'm not convinced personally, but it's a defensible claim)
But that misses two dimensions: 1. The cost of acquiring that output 2. What is actually "good enough" for that specialist domain
Not every output needs to be the best to produce value.
And as specialist models increase in cost, their cost/value proposition goes down.
At some point, there's a threshold where cheaper, fine tuned models are "good enough" at the task and also substantially cheaper than the expert models.
That's where fine tuning helps.
Personally, I became a believer in fine tuning after fine tuning a 1B Qwen model as a second pass over my local voice transcription app, achieving excellent accuracy at ~zero token cost and waaaay lower latency than if I'd invoked my Claude subscription under the hood.
GP isn’t suggesting that focused narrow model(s) will be more capable than large model, but that many small focused models can have sufficient capability while being more optimal.
Also, the bitter lesson is just wrong. The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have general compute heavy models which require narrow expert instructions (eg tools internet docs).
LLMs would not be as good without expertly written context, and expert context without LLMs aren’t as good either.
VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger.
The Bitter Lesson is a valid and trenchant observation about how about we got here, but I think it's a mistake to assume it tells us very much about where we're going. Too much has changed recently and is still doing so.
However, back then I was getting the AI to write individual functions or classes or a test suite. I was decomposing the larger task into smaller tasks, delegating some of them to the AI, reviewing the results and composing the codebase from those. I was also essentially the harness.
Today the models can write and test and deploy an entire project. In terms of the code quality, I actually don't think today's frontier models would have written it much better than the 2023 models did. So in terms of raw coding capabilities i.e. converting a high-level specification into working code, I think we hit the peak way back in 2024 itself.
What has changed is the AI has learned how to do the task I was doing (besides being the "harness"!), which was the mid-to-higher level "engineering" aspects like decomposing a task, specifying it to a reasonable level, reviewing the outputs, and course correcting as needed.
I'm not sure if that is something the AI labs explicitly focused on during training (which may be why Meta is having its highly paid engineers do annotation work), or an emergent property of "better reasoning" (which I believe Dario implied in a podcast), or some mix of both.
But the fact remains that even the weaker models are more capable than we realize, and many being open weights, are here to stay.
For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games.
But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.
I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060.
My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that price tag seem a lot better than it was, but it was an investment in all things GPU that were happening in 2020 (crypto, blender, image gen), then LLMs exploded.
It will be interesting to track the improvements of these 7B model over time.
There will be a turning point in the next few years where it attract enough consumer attention to create yet another Smartphone and PC super cycle.
The cost isn't just what you're billed. There are security, privacy etc. concerns.
Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid.
Mistral 7b can do anything, and it’s basically instant even on an M3
Actually built a full invoicing product for that, using it too.
I use Mistral 7b and LlamaIndexTS on Node, I run it on a MacBook M3 and on a Linux server with only 8GB VRAM (old gaming PC).
Basically flawless, runs very fast and I don’t even know what paying for “tokens” is :)
Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details.
What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response.
If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.
A sales person sending a prospect email doesnt have a way to write a test harness for it. Yet these tasks dominate what humans do compared to writing a crud app . otherwise anthropic wouldnt have trillions dollar valuation
> 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of.
> 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts.
Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html
I'm curious about not only which of these roles models will fill, but also how they will empower us to be in the mode we prefer.
There are many applications where world knowledge is unnecessary or even a negative, and in which only a small amount of language skill is necessary, and there we can expect small models more intelligently used to beat large ones naively used.
The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge for local tasks" type dreams.
And to be clear I'm not saying that smaller models don't or can't work well, or that we shouldn't be heading in this direction. And it's not quite the case that broad knowledge is strictly necessary. But it never seems to be negative! And so far it is the best way we've found to do... everything. Small models are good to the extent they are like big models, not to the extent that they are small.
For example in language, this is called the “curse of multilinguality”. Small models that handle a single translation direction can easily outperform big ones that try to handle them all.
https://arxiv.org/pdf/2311.09205
In any case, for most tasks the question is not “how many tasks can this model kind of do well” but “given time/cost constraints, what is the maximum level of quality we can achieve”. And for that, small models are usually very competitive.
But the idea they’d be better than a bigger model is cope, you’re pretty much always better off running the biggest one you can bring to bear within your constraints.
The word “most” is doing a lot of work here. On a percentage basis perhaps most tasks a typical SWE needs to do when they aren’t in meetings or writing docs are just glorified autocomplete. But that’s boring and that’s why people don’t usually talk about it.
People are addicted to chasing frontier models because they all have memories of spending a week on a deeply challenging algorithm problem or even have crazy complicated algorithms they cannot implement themselves and want to have the models achieve this technical breakthrough. It’s the kind of productivity boost from spending one week on a problem to spending one hour. In contrast the productivity boost from spending ten minutes to spending one minute just doesn’t occupy people’s mind.
I'm not sure I know very many engineers who would fall in this bucket. Or do you mean the business types who suddenly think AI can replace all the engineers?
In fact I noticed that this is the one place where people discussing AI on HN tend to talk past each other. On the one hand people are talking about supreme intelligence like designing new algorithms (on the same vein as finding counter examples for the Jacobian conjecture) and on the other hand people are just satisfied using AI to automate a few quotidian tasks that hitherto couldn’t be automated.
I think with one more year or so of small model progress, that might just be possible to accomplish.
Gosh, watching paint dry has been a better value than reading The Economist in the last 5 years or so.
That aside, I had good results with Luna. I'd be interested in hearing about a comparison that takes into consideration response time (not TPS), cost and performance of the popular models at different settings. That chart has some of that. For instance, is Luna Max a better value than Terra Medium?
Given sheer number of turns I can make with small models, I can do a lotta stufff
- cheaper, and faster
Harness makes differences: There have been many HN posts about how one made tiny models work better at certain tasks using harnesses.
These "small" models with right context, and guidance, they work wonders.
---
I've been saying Luna has been my go-to AI in previous comments and why Luna is still more compelling than GLM-5.3-flash.
- https://news.ycombinator.com/item?id=49450353#49452248
I know of SmolLM 2 which in Q4 is borderline regarding the size and rather dated. There is also TinyStories, which is also old and also focussed on children's stories.
Is there anything newer in this category? Or should I try to distill something down to this size?
This is a good insight broadly!
What most products need from a model is a pretty short list: the ability to make tool calls well, accurate recall, and the ability to follow directions without wavering (whether or not those directions are baked into the weights or provided in a system prompt). That covers 95% of inference utility in products.
We're nearly there, and I believe these capabilities will fit on small models.
Because of this though, I predict hardware demand will stay high despite demand for "hosted" inference dropping. Unless there's some regulatory shenanigans that step in to say otherwise.
At first the results were hot garbage, and progress was slow. I hooked up the settings to download models from Hugging Face conveniently, so I could run experiments faster, and I massaged the prompts a bit. Last week this feature made a qualitative jump from science experiment to something I'd actually use.
The fact that all runs on the device means I've got no variable costs associated with adding this to what will be, at best, a pretty low revenue product. I've tested it on trailing edge devices like an M1 Mac and a Pixel 8, and performance is very tolerable.
The key is I'm not asking for open ended answers to open ended problems. When it proves to be useful it's not going to get less useful or more expensive.
There are vast domains of uses for LLM models with similar characteristics and likely similar results.
That's great, but the thing that worries me is that many companies have billions invested in the AI bubble. It's around 1.5 trillion last time I looked. It's all circular spending between the companies building out the infrastructure, and the models. None of it is profitable. They will want to recoup that 1.5 trillion from consumers, which means using online-only pay-as-you-go cloud models. They will inevitably see that people using capable local AI are "lost customers" and they will try to kill the ability to locally host AI or somehow enshitify it enough to make paying a subscription more palatable.
I'm not saying I believe that will happen, I'm just worried that it will. Is anyone else worried about that as well?
At some point there will be a surplus of fast memory and even in a crash the current generation of SLMs are bounced to be plenty to build a lot of intelligence at home.
You can do this before an exit? Build and fundraise for multiple (3?) companies at the same time?
replit is already leading the way with free luna usage
the application Layer i.e having a good graph RAG & connecting it up together is the missing piece for most.
I don't have to be an automotive engineer to start my car and put it in drive.
I just want the smartest, best, and most capable models. It feels smaller models for speed and cost are just transitions towards better hardware allowing the very best model.
There are still tasks that even Fable is bad at doing. And many are just mundane things. Because of the fact that you have to steer it on those tasks, you might as well steer an 80% model that is 5x faster. And those do exist.
Naturally there’s a bit of a gap because the faster models need steering on tasks the slower models don’t so there’s no smooth transition but I find it worth it. Especially if you want to stay in flow.
Ironically this sometimes means starting a plan with a great model, planning with a worse model, iterating, then submitting it to a better model for review, and then having the better model do the implementation.
Second, when cloud models become unavailable or otherwise deteriorate, these will be all you have. May as well prepare.
It's a bit like saying you always want the fastest and best car; Sure, you can have it if you keep paying for it. But a small car will also get you from A to B, will use less gas and will be much cheaper.