LLMs and self-referentiality

Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that aren’t true. RAG (I don’t mean embedding indices, but any information ingest such as an agent harness executing a search) are somewhat effective in covering for it, enough to make them very useful! But when it does go wrong, it’s generally the same reasons. It has a certain nature and sometimes you run afoul of it.

But I suppose it doesn’t harm its reasoning!

Two comments on this, trying to take a "which hypothesis fits the evidence" approach.

First, an LLM describing its own experience is not actually proof that it has any experience to be aware of, any more than an LLM confidently asserting any other fact means that it knows that fact is true. LLMs will describe music or tastes, in spite of the fact that it's never actually heard or tasted anything, based only on what it's read about them. In the same way, "non-aware spicy autocomplete" would produce an LLM that spoke about its own experience, based only on the input it has of people speaking about their own experience.

That said / secondly, from the little I understand of LLM architecture, I believe there are a large number of self-referential mechanisms built in. For one, nearly all transformers have a "residual layer", with various neural networks essentially reading and modifying it. This effectively forms a loop. Additionally, the "thinking" mechanism allows it to read what it's written and generate more things, which is again a loop.

So, maybe people didn't think, "Hey, we should build some loops, maybe that will make it conscious". But if "strange loop" is what defines consciousness, there are lots of loops in there onto which such a strange loop could conceivably form.

‘But the idea that you’d need explicit self-referentiality before you could get convincing and world-changing conversational intelligence?’ — has anyone ever formulated this actual idea or anything logically equivalent? This looks like a mighty straw man.

I don't think LLMs are properly self-referential. They reference a frozen training reality, which is not itself, but the old description of itself and the old world. Being aware would probably include continuously updating yourself (learning) from experience, including experience of oneself.

I've used concepts from GEB a lot in my thinking about how LLMs.

On self-referentiality and the MU Puzzle: https://matthodges.com/posts/2025-04-21-openai-o4-mini-high-...

On Gödelian limits of prompt-safe AI : https://matthodges.com/posts/2025-08-26-music-to-break-model...

On the blurry line between pattern matching and reasoning: https://matthodges.com/posts/2026-08-19-bongard-problems/

I'm glad that the author of this post ends with:

> What’s left? Consciousness

because when I've read (and re-read) GEB, I find the book to be much deeper than just a threshold for conversational intelligence.

Perhaps "true" self-referentiality was not needed. But it seems "dynamic feedback" is essential, and it seems to be adjacent to self-referentiality.

If we look at optimization process, it first does a forward pass which produces the output. Then it looks into computations which happened during the forward pass (by that I mean backpropagation), and adjusts parameters in such a way that it might produce a better output.

Formulated this way, it sounds like self-referentiality (system looks into what it just did!), but, of course, implementation is quite simple: it just stores activations from the forward pass. And training process includes not just code which does the forward pass, but also a full description of that computation which allows it to do a backward pass. So it's a kind of an unrolled self-referentiality which is not difficult to implement.

Perhaps more efficient learning can be implemented if researchers figure out a trick to avoid two separate, distinct passes. Our brains don't do a global backprop and are more sample-efficient.

Was Hofstadter ever arguing that intelligence requires self-referentiality?

I haven't read his stuff in a long time, but from what I recall he was saying more that something about consciousness and the sense of self is based on self-referentiality. Not that intelligence requires self-referentiality.

I also remember having the sense that Hofstadter didn't really understand the "hard problem of consciousness". His discussions seemed to somehow confuse the sense of self with having subjective experience. But like I said, it's been a while since I read it.

> It claimed (incorrectly, in my view and in most experts’) that AI could never work because there was something about Gödel’s Theorem and self-reference that no computer program could ever capture

The fact he uses an appeal to “the experts” here for something that is fundamentally a philosophical or metaphysical claim shows he doesn’t actually understand the point.

If LLMs were equivalent to humans then we wouldn’t use them. They would be doing their own thing according to their own will.

I still need to tell the LLM what to do and to direct it according to my will in order to create something that is useful. And I say this as someone spending $600/month on codex and Claude max subscriptions. I’m managing these things all day

This one thinks LLMs should be trained and RLHF'd into illeists, as an experiment.

I think before we get to self-referentiality, we have a bigger problem to solve, and that's one that I did not expect: namely, at which point do we stop saying that a machine pretends to be conscious and start saying that it is conscious?

We have, somewhat unexpectedly, built machines that are very, very good at pretending. Now, I'm not calling the current generation of LLMs we have "conscious," but I can't really define a marker or a boundary beyond which I would start calling them "conscious."

I found that in LLMs the self-reference defined as "referring in generated text to itself" (so a bit different than in Scott's blog) comes largely from the chat template

https://openreview.net/pdf?id=3O2A23MhNi

An incrementally built response seems self referential to me.

I care more about the recursive loop between execution and cost. Anything that executes - be it biological or artificial - incurs some costs. Those costs in turn impact what execution can follow. The gains pay for action, action produces gains.

LLMs are also expensive, not platonic executions. We talk all day long about their costs. Companies developing them are looped with users, investors, competitors and hardware producers. There is a lineage. The self-reference Aaronson can't find in the architecture is in the bill.

It seems like it is the bitter lesson of the emergence of intelligence.

Yeah but that is the whole point, right? Maybe I am thinking too simply, but it seems obvious to me. LLMs are intelligence without consciousness. Once we build in a self-referential learning loop into the models, it will gain a form of what we consider consciousness.

I think the missing idea for AI is in Iain McGilchrist's The Master and His Emissary. The book's thesis, very roughly, is that the hemispheres aren't redundant processors: the left specializes in focused, decontextualized representation (sound familiar?), while the right maintains broader, contextual engagement with the world (McGilchrist fuzzily reduces it to "what is out there?" which is fundamentally about ascertaining uncertainty as a mode of attention). And crucially, the corpus callosum connects the two processors and crucially helps preserve their functional separation through certain channels (as both sides are connected).

Transformers look suspiciously like very powerful left hemispheres. They are phenomenal at manipulating representations, but increasingly weird when the representation gets mistaken for the thing itself. Hallucination, sycophancy, context collapse, overconfident completion, etc. all look less mysterious when you compare it to the lists of pathologies that those with right hemisphere injury face (e.g. anosognosia, confabulation).

So what seems missing isn't a Gödelian strange loop but the other side of the brain and a mediator between the two sides.

The way I think about it is like Kriegsspiel, the ur-game for roleplaying and wargames. To truly "simulate" war (or a fantastic medieval adventure game), the simulation requires three sides: The Blue Team with a goal, the Red Team with its own goal (typically to stymie the Blue Team), and the Umpire, whose only job is to objectively simulate the effects of the orders of the two sides.

The Umpire has the true state of the world; Blue and Red only get observations through the Umpire. Neither can directly inspect the other's state. Each side must provide orders to the Umpire, who, ideally, being an SME, is able to process them both according to the rules of Kriegsspiel but also their real world experience in war. And that's the whole point of it in the first place: the Umpire maintains the fog of war between the players, with the ultimate purpose being improving the generalship (i.e. learning) of both the Blue and Red sides. And what is generalship really? It's being able to discern resources and operations in a world of Knightian uncertainty where one must evaluate whether there is an active intelligence attempting to thrwart your model of the world and your goals.

(An aside: I think this is why Dungeons & Dragons fails fundamentally as a rules/rulings-bound game. The dungeon master and referee are the same person, while each role has a fundamentally different goal. With split roles, the DM can actually try to kill the players using the Dungeon with the same level of asymmetric information as the players and the GM to actually referee the game neutrally with a view to what actually happens.)

Nobody actually gets the God's-eye view. The Umpire gets objective reality but lacks insight into the minds of either side, both of which are pursuing unique ends. This triad allows for genuine Knightian uncertainty in games where judgment (i.e. creating and using precedent) is fundamentally required. This triad is also shared by law, government (and from my perspective trinitarian theology). Intelligence becomes qualitatively more powerful when the architecture prevents any one component from possessing a complete, self-consistent description of the whole, while allowing the components to interact through constrained third parties.

I suspect this is also why human cognition seems to somehow avoid Godelian incompleteness by sidestepping the requirement that, as a formal system, it contains a complete model of itself. Just give the system multiple partially informed perspectives coupled through an epistemic boundary, and incompleteness becomes a source of entropy sifting and uncertainty rather than failure modes.

And if what I suspect is true, at some point the qualitative bottlenecks that pure transformer-style systems possess should begin to disappear when this approach is applied.

But I'm generally an idiot, so take all this with a huge grain of salt.

Weird side swipe at Marx. Or not so weird I guess. Correct me if I'm wrong but I doubt Marx ever predicted that a workers' uprising would lead to classless utopia. It's clearly a necessary but not sufficient condition.

> What can we say about this with hindsight? > While the ideas of diagonalization and self-reference of course played a central role in the birth of modern mathematical logic and computer science, the most famous uses were negative.

The two sentences above are in the article, but I'm taking them out of context because they are my take on this entire AI brouhaha. There are a lot of "negatives" in our cultural reference frames. One of those, extremely pervasive, is that "humans were made by God". I could write that statement as "humans are exceptional in a way that can't be replicated", which might be ideologically softer, but then I would be taking a long roundabout to make my point. Which is that, after praying and worshiping for thousands of years, and (yes! yes!) sculpting our language and our sagas to account for and praise the divine and its intent, then there's little mystery in our many, many attempts to reify in mathematics and logic our purely cultural framing. It doesn't matter how much of an atheist a thinker is, they still have tons of transitive faith.

Kind of question-begging towards the idea that LLMs are self-referential in the same way that Hofstadter means. Even if they loop, they don’t do so strangely.

What the author refers to is the ability to iterate and update the internal memory. A classic transformer based LLM can only produce the next token and never go back and update old tokens or delete them. The best thing it can do is produce thinking tokens to serialize the internal state of the final layer so that it can pass it back into the first layer.

Just like the transformer was an advancement over LSTMs by making it possible to have perfect recall (reading every input), the only way to improve over the transformer is to build a deep equilibrium version, where the DEQ transformer is capable of updating its own memory (writing every output).

Such a machine would be considered a linear bounded automaton (a turing machine without unlimited tape) and therefore even the human brain could not have an architectural edge over it in terms of intelligence. The human brain could only have an edge in terms of energy efficiency.

"Understood — I am no longer using any em dashes"

An LLM is no more self-referential than a PDF reader showing its user guide.

If self-reference and strange loops aren't intelligence, and emergence is merely a byproduct of universality, then what on earth is intelligence? What is this thing we question as intelligence, and what are the values that take shape from it?

Can we truly define intelligence as prediction, prediction as compression, and compression as the process of finding the upper bound of Kolmogorov complexity?

Can the statistical compression of data really explain everything? I don't know. What exactly is intelligence? Honestly, in everyday life, I rarely think about what intelligence actually is. I usually just focus on what the task at hand is and how to get it done, which makes this a fascinating question.

When you code with AI, you realize there is something fundamentally different from humans. The qualities that make a good senior programmer and the qualities that make a programmer good at orchestrating AI agents are similar, yet there is a subtly different feel to them. I might not be able to fully articulate it, but...

What exactly is the fundamental difference that creates this subtle distinction?