Rich Sutton on AI creativity and discovery

But I think humans are better at it, while ML is better at algorithmic thinking. “Better” being more efficient and something we more enjoy doing; we can also more accurately judge what subjectively appeals to humans (i.e. taste).

I think ML should be optimized for tasks that require more generalization than programming, but are still mostly logic. Like software development, translation, and tools for art and discovery.

AlphaGO is given a hard evaluation externally. It did not itself come up with it.

When GAI models are given an external hard evaluation, they can also succeed in many different domains (that is one of the remarkable features, succeeding in many domains) ranging from simple programming tasks to frontier mathematics (disproving conjectures recently) to writing more optimized kernel code than before.

And there is plenty of RL especially in these fields where the solution may be extremely complex but eval is rather less complex. And even the discovery and the "evolution-like" trace-selection is also happening.

For this reason it seems strange to compare it to AlphaGO as alphago is given a hard eval independent of itself, from an external source (humans) in a narrow domain. If GAI is given such, it can also show some remarkable results.

But what I find more strange is that innovation and moving forward in many many many cases does not require truly novel ideas but instead a high-quality execution of layering different methods, tactics, ideas on top of each other. Because in many domains our collective knowledge is incredibly sparse and complex, something being able to recombine tools, models, ideas in a high quality way (as he mentions being selective) I think is extraordinarily powerful. And in such cases, with a finite exploration horizon (time, resource available) with 1% "good choices" vs 3% "good choices" are worlds apart, incomparable.

Most importantly: none of the above is about intelligence, it's barren solution-farming to important, valuable problems we have. Most of the AGI and intelligence-related debate seems to miss out on this simple fact. (Insert the usual stuff like a plane being unable to fly like a bird or a submarine not swimming is totally irrelevant to it being useful).

And then a final point: do we really think this thing is incapable of doing better on average on problems we average people face in our lifetime? What should we think, how should we define human intelligence when we give out degrees in science or medicine for 60-70% exam results on problems considered to be generic in the field?

If it's a), he doesn't propose such an algorithm, and I don't know how you'd do it at such a low level because how do you quantify abstract goals? Did he suggest such an algorithm and I misread? If it's b), that already exists, see AlphaEvolve or any number of things he said. Or, to be a bit of a smart-ass, just type /goal and let it rip ...

I also think he's just categorically wrong that LLMs cannot do good and novel things. And if it can, then you could just say "well that's not novel, that's derivative". A simple example, if I make up a programming language with an LLM and it works well for my purposes, then is that not novel and good? I mean, is any language other than FORTRAN not novel?

Everything is derivative and you can put an LLM in a loop to evaluate LLMs trying things. I must be misunderstanding because he's too smart to be this wrong.

That contradiction kind of says he doesn't know what he's talking about.

Should we automate exercise and play as well? How about learning?

The machine didn't have a soul, so we donated ours.

Eureka! My AI found it!

Best thing about nerds is watching them try and build frameworks and formulas for the creative act. Like a metronome trying to compose a symphony.

I don't think I would attribute anything in that process that I would consider an AI to be incapable of.

The characterisation of variation like this would seem to rest on the same 'random but directed' crutch that some free will arguments rest upon.

There is no random but directed of course, there is random and there is caused, and there are things that use both as components, but the random remains wholly random, and the caused remains entirely deterministic.

I think there is a good case to say that, in many fields, AI is better than humans at evaluation.

To find avenues to consider, I'm not entirely convinced that human innovation is more than a heuristic that appears more chaotic by virtue of a inconsistent and opaque formulation.

Many aspects of ideas com from noting how some two things are different and then considering that axis of difference when applied to another thing.

The possibilities thrown up by this extremely simple method are vast enough to require multiple layers of evaluation, most could be dismissed out of hand by a quick 'This is nonsense' check that I suspect people do so often and at a rate that it wouldn't even rise to the level of consciousness.

His main point is that discoveries involve

1. Variation,

2. Evaluation, and

3. Selective retention.

He makes a jump saying AI is only capable of 1) and humans are capable of 1) 2) and 3). I don't know what makes humans special enough that they can do 2) and 3)?

In fact, the more you think of this it is kind of strange - in science humans can only do "evaluation" because they have access to the real world. They can evaluate a new drug because they can do it on people so it is not some inherent limitation of AI but rather access to physical realm.

Finally I want to ask a specific thing: how do you mathematically falsify what this person is saying? How can you formally prove that - no AI can not "evaluate"? I ask because I make AI evaluate a lot of people's claims and it works for me.