"AI" will never become conscious

I think the whole article is undermined by this paragraph:

> We don't know what consciousness is. We don't know why we have it. We don't know why some animals have it and others don't. We don't know if it can be quantified. We don't have a model of consciousness or know if one can be created.

I agree that we don't know what consciousness is and can't quantify it, and yet the rest of the article is… a quantification of consciousness as a human evolutionary trait that cannot emerge through genetic algorithms or LLM training.

Genetic algorithms certainly differ from human evolution in numerous ways, but I don't agree with the fundamental distinction the article makes—that evolutionary algorithms are constrained while human evolution is not. The 2D cars can't "hack the simulation and turn the gravity down" any more than humans can hack the laws of physics and turn the gravity on Earth down. Yet humans attained consciousness.

I do not know whether LLMs are conscious or not, but I do firmly believe it's possible to build a machine that is conscious: after all, humans are conscious and, unless one subscribes to the kind of religious thinking the article derides, a perfect simulation of the physical workings of a human would thus also be conscious. To my knowledge, nothing we've seen so far in physics rules out such a simulation one day existing.

I also suspect that, once we do manage to make a machine that is conscious, we'll spend inordinate time and effort trying to convince ourselves it's not in order to justify the ways we treat it.

It would be helpful to define "consciousness". I'm not convinced that i as a human have this "consciousness" thing. In fact, I strongly suspect that i am merely a statistical machine picking words from a learned distribution.

But I don't want Claude to become conscious. I want Claude to help me write good code.

I don't disagree with the title of the article but I also don't think you can prove anything about consciousness with language. How consciousness can arise from matter is still a very much a mystery.

> To evolve consciousness, an LLM would have to be programmed to evolve consciousness, and to do that, we would need to know what consciousness is and how to program it.

Since consciousness only counts when it is formed deliberately rather than accidentally. Though that would mean…

How is it that sign language association learning is different from spoken language learning? What about humans using simple sign language in their early life?

Is it not possible that we try to detract too much here?