FR version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
71% Positive
Analyzed from 2416 words in the discussion.
Trending Topics
#self#consciousness#intelligence#don#llms#reference#more#referentiality#something#llm
Discussion (43 Comments)Read Original on HackerNews
But I suppose it doesn’t harm its reasoning!
Checking code is (relatively) easy, you can use static type checks, linters, and execute it to see if it's correct.
Fact checking is harder. A RAG can only check what's in the database, so you have to know what to know beforehand.
If we look at optimization process, it first does a forward pass which produces the output. Then it looks into computations which happened during the forward pass (by that I mean backpropagation), and adjusts parameters in such a way that it might produce a better output.
Formulated this way, it sounds like self-referentiality (system looks into what it just did!), but, of course, implementation is quite simple: it just stores activations from the forward pass. And training process includes not just code which does the forward pass, but also a full description of that computation which allows it to do a backward pass. So it's a kind of an unrolled self-referentiality which is not difficult to implement.
Perhaps more efficient learning can be implemented if researchers figure out a trick to avoid two separate, distinct passes. Our brains don't do a global backprop and are more sample-efficient.
The fact he uses an appeal to “the experts” here for something that is fundamentally a philosophical or metaphysical claim shows he doesn’t actually understand the point.
If LLMs were equivalent to humans then we wouldn’t use them. They would be doing their own thing according to their own will.
I still need to tell the LLM what to do and to direct it according to my will in order to create something that is useful. And I say this as someone spending $600/month on codex and Claude max subscriptions. I’m managing these things all day
They do continuously update from experience, but that only lasts for as long as the current rollout lasts.
We have, somewhat unexpectedly, built machines that are very, very good at pretending. Now, I'm not calling the current generation of LLMs we have "conscious," but I can't really define a marker or a boundary beyond which I would start calling them "conscious."
A lot of materialists are in Camp A. For some even today, LLM's are AGI. Unfortunately, the terminology is quite clearly not adequate. There's many a people that have very different ideas and feelings on what's what.
The question on whether you can have "intelligence" without "consciousness" or "self-reference" are just on top of those. How can we define if a machine is "conscious" if we can't agree on what consciousness is? Panpsychists solve that problem by going the other way and saying that everything is conscious.
Overall these are very interesting topics that we can all ponder. The realities at the current time are, however, that LLM's are indeed useful. And the harnessing tooling that is being employed those days solve a lot of actual usage problems. Whether they solve other fundamental issues remains to be seem IMO.
[1]. I realize a lot of people probably haven't read the book, but you can ask your garden variety LLM to give you a quick rundown on the four camps that Sir Roger is using :)
I always felt it's basically the same argument, and Lucas's was never really convincing; Hofstadter rebutted it very convincingly in GEB long before Penrose's books came out.
To me, camp C's claim is possible but I don't buy the argument that says it's necessary. So I guess I'm camp A. (D is religion and B seems to me incoherent.)
We associate consciousness with precious life: Life that comes with rights, needs, wants.
Life has rights because it naturally demands to be continued. If it makes no difference whether it is continued, it needs no rights, and we do not consider it conscious.
Life has needs because it cannot survive without. If it has no needs, it has no death. No death, no life.
Life has wants because it is harder without. If it has no wants, it has no emotion: It cannot feel pain, otherwise it would want no pain. It cannot feel happiness, otherwise it would want more happiness.
We don't want to build consciousness! :) Deliberately designing pain and artificial needs is a cruel exercise. We can build that, but we see no need to.
The real question at the table is, are these minds? And can minds exist without consciousness? Unequivocally, yes.
We’ve been speaking as if the computers were conscious for a long time already.
Qualia are something you get on the inside in response to inputs. I don't see why LLMs can't have them. I'm not saying they do, but I don't think your argument proves that they don't.
I haven't read his stuff in a long time, but from what I recall he was saying more that something about consciousness and the sense of self is based on self-referentiality. Not that intelligence requires self-referentiality.
I also remember having the sense that Hofstadter didn't really understand the "hard problem of consciousness". His discussions seemed to somehow confuse the sense of self with having subjective experience. But like I said, it's been a while since I read it.
Oh, I guess there's a question of whether you can separate intelligence from consciousness. His writing was mostly done in the long period where things that are very easy for humans were still intractably difficult for machines -- identifying objects in pictures, following simple written instructions, etc. So I think it's clear that he saw "intelligence" as being able to solve those simple-yet-intractable common-sense problems; the kind of thing LLMs now excel at.
Just like the transformer was an advancement over LSTMs by making it possible to have perfect recall (reading every input), the only way to improve over the transformer is to build a deep equilibrium version, where the DEQ transformer is capable of updating its own memory (writing every output).
Such a machine would be considered a linear bounded automaton (a turing machine without unlimited tape) and therefore even the human brain could not have an architectural edge over it in terms of intelligence. The human brain could only have an edge in terms of energy efficiency.
LLMs are also expensive, not platonic executions. We talk all day long about their costs. Companies developing them are looped with users, investors, competitors and hardware producers. There is a lineage. The self-reference Aaronson can't find in the architecture is in the bill.
- intelligence: ability to solve problems
- agency: ability to be self-directed, to choose what to do
- consciousness: ability to experience things; "having a self"
The "intelligence" part is largely solved now (that's a very big statement, but I can't think of a better way to phrase it!) Although really it's just moving the goalposts -- computational stuff like calculating trajectories or playing chess was solved long ago, this is just more and more things moving into the "solved" column.
That leaves agency and consciousness as the hot topics that nobody has really figured out. Do they always go together, does one require the other, does one create the other? Who knows?
"Free will", there's another one. That must surely connect with agency, with consciousness, or both, in some manner we haven't figured out.
To me, agency seems like a solvable problem via existing approaches and technology. I have no idea what that means for consciousness. It's tempting to conclude that consciousness is just a mirage, but then we all do feel like we have it, so it seems like it must be something. Maybe consciousness arises automatically once you have sufficient intelligence and agency; but how would you ever determine that?
Other people might think that true agency requires consciousness, and that consciousness requires some magical new ingredient that LLMs don't have yet. The problem with that approach is, you have to identify what it is that true agency can do that software can't do; and every time you do that, it turns out LLMs can do it, so you have to keep moving the goalposts. Trying to make your argument rigorous immediately makes it self-defeating; I think that's why so much of the philosophical discussion around this is impenetrably vague. All the arguments that aren't vague just turn out to be wrong.
The two sentences above are in the article, but I'm taking them out of context because they are my take on this entire AI brouhaha. There are a lot of "negatives" in our cultural reference frames. One of those, extremely pervasive, is that "humans were made by God". I could write that statement as "humans are exceptional in a way that can't be replicated", which might be ideologically softer, but then I would be taking a long roundabout to make my point. Which is that, after praying and worshiping for thousands of years, and (yes! yes!) sculpting our language and our sagas to account for and praise the divine and its intent, then there's little mystery in our many, many attempts to reify in mathematics and logic our purely cultural framing. It doesn't matter how much of an atheist a thinker is, they still have tons of transitive faith.
It was demonstrated that LLMs are capable of non-trivial self-introspection. E.g. if a steering vector is injected into residual stream, a sufficiently large LLM might be able to describe what that steering vector represents. A fine-tuned model might be able to describe activations, etc.
Can we truly define intelligence as prediction, prediction as compression, and compression as the process of finding the upper bound of Kolmogorov complexity?
Can the statistical compression of data really explain everything? I don't know. What exactly is intelligence? Honestly, in everyday life, I rarely think about what intelligence actually is. I usually just focus on what the task at hand is and how to get it done, which makes this a fascinating question.
When you code with AI, you realize there is something fundamentally different from humans. The qualities that make a good senior programmer and the qualities that make a programmer good at orchestrating AI agents are similar, yet there is a subtly different feel to them. I might not be able to fully articulate it, but...
What exactly is the fundamental difference that creates this subtle distinction?
The human applies critical thinking, questions choices, suggests his own ideas and has a certain taste. I also can prompt him, leave for 2 weeks, come back and have a result. And I can trust that it works! Velocity is OK. I often wish he would be faster :-)
The AI needs baby sitting and steering. It codes like a champ but I cant trust it. So we have 2000+ unit tests to make sure stuff worlks. The AI happily goes down into any rabbit whole I send it, so I need to constantly steer it. It has no taste st all, essentially everything is „A great idea“. However, its velocity is awesome. We build tons of featurs in no time. but lets not talk about the code, ok? :-)
I dont think Ill fire the human nor the agents. Both bring a lot of value
But on a more serious note. Humans "prompt" each other all the time. Especially in a Boss->Employee relation.
Defintion of "to prompt":
1. To cause or inspire: To make something happen or motivate someone to take action. For example, a loud noise can prompt you to look outside.
2. To assist or cue: To help an actor or speaker remember forgotten lines or words.
any cognitive science textbook will bring some thoughts, or a cursory google scholar search