Overtraining as the path to human-like AI

The goal feels underspecified. I'm not sure I fully understand the perceived gaps between frontier LLMs and AGI. I read Gwern's original article and it was written two years ago. Several statements it makes were true at the time but no longer are.

In particular I'm not sure the statements about models making bizarre mistakes still holds at this point. It's been a long time since I saw good models do something that seemed genuinely stupid or strange. In the rare cases it happens it can be easily explained by aspects of the algorithms e.g. the model can't erase an already committed token and has to always build on it. I think a big part of why hidden reasoning helps is it gives the model a place to draft an answer where mistaken tokens can be ignored, hence why they're full of "Wait, but..." style tokens.

This leaves the (rather vague) inability to generalize. Is that true? How would one benchmark this? My perception is that LLMs are now superhumanly intelligent in nearly all ways, with exceptions missing only for continual learning (with the latest memory/note taking features, even this is arguable), and perhaps some very vague inability to have "shower thoughts" and "innovate" via non-obvious connections between things. But I see no reason why that is fundamental and the progress in maths proofs suggests it's not.

In other words, the notion that we need to massively increase param count might have sounded good in 2024 but seems kinda weird and pointless in 2026. What's the expected outcome? Again and again what I hear from colleagues and experience myself is that we're not really intelligence constrained at this point. Smarter models aren't going to fundamentally change how we use them. The roadmap looks more like exploring the cost/benefit landscape to figure out where AI should be applied and when not. That meta-work is hard to automate because the landscape is covered in a fog of war with super sparse rewards, and good results tend to come from intuition, experience and having contrarian opinions that turn out to be right.

My guess is human learning requires some major differences in training.

For instance, we learn to process a firehose of visual, sound, tactile and motor information, before we start thinking in well-defined concepts.

That is a major foundation, grounding, and highly organized state, on which we learn language. And is entirely missing for LLMs.

We learn a lot metaphorically, so it may be that the physical effects we learn to recognize act as highly efficient sub-vocabulary, for quickly learning new concepts both verbal and nonverbal.

So a series of different learning stages, each making the next more efficient, is likely to be a big part of any solution.

No amount of data/exposure will help a baby trade stocks. General fast learning requires other things to be learned first.

No.

1. "Grokking" was shown on 4-digit modular arithmetic with a 1-layer transformer; this article extrapolates it to AGI and a $10B training run with exactly zero intermediate evidence.

2. The "small dataset" is 25 trillion tokens - literally the size of current frontier training sets - but calling it small sounds revolutionary.

3. BabyLM has spent 4 years failing to produce grokking on constrained data; the paper gets a footnote saying "those models were too small," which is unfalsifiable until someone burns $10B.

4. Chain-of-thought is already empirically required for frontier performance - it's expensive, bizarre, and nobody predicted it - yet somehow we're supposed to bet the farm on a phenomenon that has never scaled past arithmetic. We need that data, even if it is just "Actually, ..."

5. If you want to chase "recurrent depth", loop transformers rumored in Mythos/Fable are at least grounded in actual engineering; grokking-at-scale is just vibes ai bro science.

More data is and will always be the answer. Why are all labs distilling from each other?