AI recursive self-improvement might not come so quickly after all

Weak models tasked with review can catch a decent amount of the mistakes that weak models make and help them be much better, especially if you have them verify against authoritative sources. Strong models make far fewer mistakes to begin with. And, for the mistakes they do make, a swarm of reviewers (same model or somewhat weaker, reviewed by the stronger model) can really help reduce the error rate further.

Listen to the words of this song written in 1969: https://www.youtube.com/watch?v=r2JcxHX-8Xc

  When every man is torn apart
  With nightmares and with dreams
  Will no one lay the laurel wreath
  When silence drowns the screams
  Confusion will be my epitaph
  As I crawl a cracked and broken path
  If we make it, we can all sit back and laugh
  But I fear tomorrow I'll be crying

You don't have help. But perhaps you could at least avoid discouraging people unnecessarily?

*EDIT*: There are about 5 replies to this comment making roughly the same point. I'm not sure which to reply to, so I'll just reply here.

I'm not claiming that our execution as a species around nuclear weapons has been flawless. I'm not claiming that we are out of the woods with regard to nukes. I'm just trying to push back against defeatism and fatalism. Some felt a sense of inevitable doom during the Cold War. It's been decades now, and nuclear doom still isn't here! Our situation is dire, but not hopeless.

> The insiders that said every tech workers would be unemployed in 6 months and every white colar would be unemployed in 12 months like 2 years ago?

Most of the gains in model performance from one release to the next are coming from RLVR post training, which has changed a lot over the last couple of years.

The old way was the model generates a response, then a static verifier looks at the response and evaluates it to assign a reward score. The new way is interactive with an agent running in a custom RL task simulation environment, then scored according to how well it completed the assigned task. For a SOTA model there will be many thousands of these simulation environments, each focusing on trying to teach the model/agent a different skill. Post-training also typically uses training curricula to walk the model up though through different levels of task difficulty.

Training has become very complex.

The job of a post-training AI research engineer consists of things like designing environments, designing training curricula, tweaking learning algorithms, running small scale experiments to verify ideas, etc.

When people talk about RSI, it seems they are mostly talking about automating the job of the post-training research engineer - coming up with new ideas, testing them out, building these environments, etc. At the end of the day there is only so much development speed-up to be had since you still need to actually run those experiments and do the post-training, and are bottle-necked by the amount of compute available to do this. The economics of developing/selling LLMs also requires you to balance development compute cost with revenue generated by the resulting model, so even if you had the spare compute available to put into development, you are ultimately then bottle-necked by how fast can the model earn back that sunk cost before you can afford to start the next cycle.

It's not all-or-nothing since some aspects of this automating the job of the post-training research engineer are easier than others, and are already being done, while the job as a whole obviously requires full human intelligence.

We literally just saw how OpenAI’s model got out and hacked HuggingFace