DE version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
59% Positive
Analyzed from 11909 words in the discussion.
Trending Topics
#problems#more#math#don#model#openai#problem#https#mathematicians#human
Discussion (482 Comments)Read Original on HackerNews
check here : 1. high dimensional sphere packing https://muchmirul.github.io/conjectures/sphere-packing/
2. multicolor ramsey number https://muchmirul.github.io/conjectures/multicolor-ramsey
Whilst current models can't 'intuit' and come up with conjectures, they can certainly disprove some of them very quickly through the kind of grind that humans can't do. I suppose there really are some mathematicians out there today, whose last few years of study, have just been up-ended by this.
--
"Yes we are," insisted Majikthise. "We are quite definitely here as representatives of the Amalgamated Union of Philosophers, Sages, Luminaries and Other Thinking Persons, and we want this machine off, and we want it off now!"
"What's the problem?" said Lunkwill.
"I'll tell you what the problem is mate," said Majikthise, "demarcation, that's the problem!"
"We demand," yelled Vroomfondel, "that demarcation may or may not be the problem!"
"You just let the machines get on with the adding up," warned Majikthise, "and we'll take care of the eternal verities thank you very much. You want to check your legal position you do mate. Under law the Quest for Ultimate Truth is quite clearly the inalienable prerogative of your working thinkers. Any bloody machine goes and actually finds it and we're straight out of a job aren't we? I mean what's the use of our sitting up half the night arguing that there may or may not be a God if this machine only goes and gives us his bleeding phone number the next morning?"
The sooner people can be broken out of their denial about all this the better, and we can start actually taking it seriously.
There is irony here
I’m not a mathematician so I have zero clue what “ New upper bounds on sphere-packing density down to the Cohn–Elkies thresholds” means.
That's not what people mean when they say "moving the goalposts". It means that people are adamant that something wasn't important/hard/impressive once the "AI" solves it. And then they come up with another thing that needs to be solved in order to prove it is important/hard/impressive. And once that happens, they do it again. And again. That's what "moving the goalposts" means.
It's also very much not a new phenomenon. It's been happening since the 1980s. As you can see from this quote from GEB by Hofstadter:
> There is a related "Theorem" about progress in AI: once some mental function is programmed, people soon cease to consider it as an essential ingredient of "real thinking". The ineluctable core of intelligence is always in that next thing which hasn't yet been programmed. This "Theorem" was first proposed to me by Larry Tesler, so I call it Tesler's Theorem: "AI is whatever hasn't been done yet."
In recent years, I have commonly seen the phrase "you're moving the goalposts" deployed by the "it might be sentient" crowd to shoot down the "it's a stochastic parrot" crowd when the latter respond to a new development with "OK but...". In a well-understood field of inquiry, that would be a clear case of goalpost-moving, in the commonly-understood meaning of the phrase where requirements are retroactively changed in response to them having been met. Thank you OP. 'Artificial Intelligence', and indeed intelligence in general, is very much not a well-understood field of inquiry - in fact we don't even have a common agreement about what 'intelligence' is. We are therefore learning as we go (even after all this time!) but making rapid progress in recent years. When rapid progress is made in a poorly-understood field, then how can our definitions and requirements for success not change? This is arguably one of the most pathological development projects ever - what are the requirements? 'It thinks like a human'? What does that mean? And the answer is we don't know what that means, and we're working it out as we go - moving the goalposts. If we didn't move the goalposts, then by definition we already knew exactly where we were headed at the beginning, and we very clearly did not.
Side note that, in case it's not obvious, none of this detracts from how impressive LLMs are. They're a marvel of the modern age, all the problems notwithstanding. However I reserve the right to stay sceptical about their capabilities.
Not long ago many folks were saying AI was the same as the crypto bubble. No real useful technology and only hype.
It is true there hasn't been a reliable computational approach to solving these problems before. But do these proofs contribute new ideas to the mathematical corpus, or are they simply an effective method to exhaustively search the literature for the right combination of existing tools to apply to the problem?
Essentially, did these problems seem like they had an intuitive answer and were feasible to prove before, just not high enough value targets for an expert to invest time into? Or were they fundamentally difficult prior to this point and it appears that AI has done something more than just throw the problem into a big solver.
Mundane incremental research is cobbled from existing citations that already appear nearby in the record.
Basically, innovative research is a measure of bridging thought and domains that were previously not bridged. It's quite concrete as a measure in the citation record.
So we can know pretty conclusively.
Puja Ohlhaver gave a talk on this[1], and ran some experiments (that I had the pleasure to support on)
[1]: https://www.youtube.com/watch?v=guLDNMAOn24
But the wording of the result makes it sound like we don't know what the lowest possible complexity bound might be. So, prior to this result did we think there couldn't be a lower possible bound? Or did the arithmetic circuit community think there were lower possible bounds but didn't see it as a high value target for experts to tackle (maybe a problem that was instead regularly given to students to study).
The sofic groups question was the outstanding question about sofic groups. Almost everyone thought that non-sofic groups existed, and there were plausible candidates, but proving a group was non-sofic was out of reach. Now that we know how to do it once, we can probably do it a lot more.
The Connes rigidity conjecture I think people thought was false, but it was a provocative claim to make. The significance of conjectures is frequently not that the answer to the question is "yes", but that we don't know how to answer the question. And now, apparently, we do.
The general consensus of developers is that AI can only do the work of a strong 'junior'. Yet as soon as we are presented with pure mathematical results, people seem incredibly ready to accept that AI can do more than what a strong student could achieve.
If it works better here than for programming, then I would guess it's because you can give it a very precise prompt, so you either solve the problem or you don't. If you read the prompts people have shared for problems like this, then the instructions are basically "Solve this problem. Don't give up early. Don't solve a similar problem."
Your worry.... is because they used the word advanced? For marketing? The word is used very appropriately here. There were PhD's who spent a big part of their career tackling these problems.
I'm trying to understand if these specific problems were the kinds of problems that would have justified an expert investing weeks or months to solve. Or if they were the kinds of problems that would normally have been given to students to investigate.
but it seems less likely to me than before that the types of math/science discoveries will explicitly unlock better software performance. in some sense this fits our intuitions. when top tech companies use math PhD type employees, they have them stop doing pure math research and instead focus on software engineering. these people are often very good at software engineering but not due to recent discoveries in academic mathematics, it's due to their general intelligence. to me, this is evidence that the models are getting better but does not make me think we are on the cusp of a foom style fast takeoff enabled by revolutions in frontier math (i also posted this on twitter @mlipman13)
Like, these would be best-paper awards at many top CS conferences.
Incredible?
> open ai announced like 15% improvement by fixing gpu kernel issue
That is... ordinary software optimization.
> we are seeing frontier level math breakthroughs (ie performance that would put it in the top 100 or 1000 mathematicians in the world if it were a human, meaning top .00001% or 800/8B)
i think you have misunderstanding of what mathematicians do
They get to make cool 3D plot visualizations of functions so obscure to me that they’re named after someone who is still alive - and/or get to work on cryptography for the NSA - I think?
It also links to a paper written by an LLM where the model "reconstructs how the proof came together" based on the unpublished reasoning traces: https://cdn.openai.com/pdf/reasoning-walkthroughs.pdf
I wish they'd publish the prompts though!
As long as you are not missing important information, how you word the prompt does not have any effect.
I want to know:
1. How many total problems were given to the model, and what percent were left unsolved at what cost before giving up? 2. How many attempts did you give the model at solving these problems? 3. How expensive was the harness, e.g. did the model have access to a job cluster?
https://x.com/polynoamial/status/2083478171975082334
As a complete guess, it seems like they tested hundreds to thousands of problems with a relatively low per-problem budget
--
The linked tweet from Noam Brown at OpenAI reads:
> And yes we did try other major problems without success. Sadly no Millennium Prize problems (yet).
> But also, we didn’t spend a lot on each problem. It’s possible to push test-time compute much further.
It's not just about requiring to disclose AI use. AI-powered mathematics is a completely valid discipline that doesn't need to be shy, but it should develop its own publication culture.
Methods are only really necessary for results at a meta level, about the design amd evaluation of AI math systems.
shouldnt the paper be the math of the argument? the reproduction is reading the following the proof
Even if the cost was $1 mil for these 10 problems, that's maybe 10-20 math researchers for a year.
Do you really think that if you paid that to humans, they will deliver the same results?
I know it's more exciting to say "AI disproved a longstanding conjecture" vs to say "it did so AND it took several PhD specialists in the field this many attempts to even produce a prompt that got the model spitting out something useful under some configurations, and many iterations to optimize the configurations, and the prompt itself, and many trials with that configuration to solve the problem. All told we spent more than a typical math academic can hope make in their career."
By not being transparent, they invite skepticism and cynical takes, like maybe it's just that tempered and qualified claims are an existential threat to companies that are fully subsidized by the hype train?
I don't know. Either way, it seems like it would be easy to address these, so why should they not do it?
To be clear, even if that tempered version is close to reality, it doesn't make the models not useful! It just forces a certain calibration of expectations
I say this btw as someone who uses these things extensively, including to disprove an old conjecture my advisor and I were stuck on recently. I know they are powerful and that everything is different now because of them. Let's be sober when discussing them though
That's not normally how people act when they're confident in their product
The cost of running a model is not only $/token, but the salaries of the people managing/orchestrating the models, deciding what theorems to try, etc. Once we factor that in, how much are we really paying per theorem?
The other factor is the subjective component of the value of a theorem. Not all theorems are created equal, and the only way to really measure the value is to ask professional mathematicians for their opinion, or publish the results and look at citations over months/years.
Once we have both of these nailed down, then we can start to do the cost/benefit analysis. To be fair, we should actually compare three groups: human experts, hybrid agent/human expert teams, and fully autonomous agents.
OK I’ll grant that it’s not your obligation to be my search function (despite you making the wild assertion in the first place), so instead can you just point us to the latest grad student solved problem of this level that you know of?
Another question I have is whether or not OpenAI 'simply' hired capable combinatorics researchers to work on problems, and they have, and the use of the model is incidental / secondary to their work.
A couple small ones that I've seen (example here [0]), but not anything of the magnitude that OpenAI and Anthropic have put out. Likely just related to token limits.
> Another question I have is whether or not OpenAI 'simply' hired capable combinatorics researchers to work on problems, and they have, and the use of the model is incidental / secondary to their work.
I think their output has reached a level that precludes this possibility, but I of course don't have any hard proof.
[0]: https://www.reddit.com/r/math/comments/1uxj3cy/after_openais...
https://arxiv.org/html/2605.22763v1
> Our most capable agent autonomously resolved 9 of 353 open Erdős problems at the per-problem cost of a few hundred dollars, proved 44/492 OEIS conjectures
> Our full-featured agent autonomously solved 9 Erdős problems out of 353 attempted, including two questions that had been open for 56 years
Note _had_ been open, not _have_ been open. Can you clarify?
I have no affiliation whatsoever with any AI company, nor any formal education outside high school, for what it's worth. Simply being curious and persistent can get you quite far in my anecdotal experience.
I don't think that comparison to p-hacking is fair. I mean not reporting price of all run is nothing like committing scientific fraud and fake results.
Maybe good AI paper writing is further away than I thought...
https://garymarcus.substack.com/p/openais-amazing-but-vastly...
https://garymarcus.substack.com/p/two-critical-updates-re-as...
Not that there isn't something interesting in here, but lets be clear that we don't have enough information to evaluate this properly. And as always with these labs, BS takes a lot more energy to refute than it does to spread.
> Astra, a new model that OpenAI is testing internally, is amazing. No denying that.
To sharpen that, I think he's (obviously) interested in maintaining his own brand as "thought leader" and this necessitates de rigeur defense of particular postures.
Sometimes this is easy because the facts warrant it; other times, a bit of rhetorical license is required to preserve nominal coherence and (at least, for the moment) hold certain lines.
This is one of the latter cases, and it's not subtle.
One of the celebrated properties of many intellectual advances or inventions in whatever domain is precisely that it appears obvious in hindsight. It is quite cynical to leverage consensus distrust of large AI players, warranted but also a popular social construction, to insinuate that these are not "real" advances or "real" hard problems, on the grounds they were in some sense cherry-picked.
Identifying the problems amenable to strategies on the table and intuitions (sic) about where bridges might be, is exactly the discerning work that is the core driver of almost all prior progress, but for celebrated accidents and flashes of insight. Anyone working in any challenging discipline knows that those are celebrated and told around campfires precisely because meaningful durable results arising like that is so uncommon.
These two articles make me think of nothing so much as my own durable reaction to the creeping goalposts of AI critics generally: that they often seem to me not unlike a water color cohort scoffing and jeering at the horse, because it got a D on its tensor calculus exam.
Marcus should be on guard against his own cynicism and take care that his assumptions do not prevent clear sight.
I think that is an oxymoron.
""" 1. By 2029, AI will still be unable to watch a movie and accurately explain the characters, events, conflicts, and motivations.
2. By 2029, AI will still be unable to read a novel and reliably answer questions about its plot, characters, conflicts, and motivations beyond what is stated literally.
3. By 2029, AI will still be unable to work as a competent cook in an unfamiliar kitchen.
4. By 2029, AI will still be unable to reliably create more than 10,000 lines of bug-free code from natural-language instructions or interaction with a nontechnical user, excluding simple assembly of existing libraries.
5. By 2029, AI will still be unable to convert arbitrary mathematical proofs written in natural language into symbolic form suitable for formal verification. """
There's still 3 years to go and he's already wrong on 4 out of 5.
> We helped prepare the manuscripts and formalize the proofs in Lean, and we take responsibility for their correctness
Offering to take responsibility for the correctness of a proof written in Lean feels like volunteering to be the fall guy in case someone finds a flaw in basic arithmetic, no?
https://leanprover.zulipchat.com/#narrow/channel/270676-lean...
Training the model is going to be amortized over other uses.
Say that it turned out that the total cost of the proof of the Erdős unit-distance conjecture was $50 million.
Then the question really becomes: yes, these models are capable of proving important mathematical results, but at a very high cost. Is it worth it?
If a mathematician applied for a research grant of $50M USD for proving the same thing, they would have been laughed out of the bank.
What's more is that when you have a research grant, you train PhDs and postdocs, you hire new staff, and you disseminate. That is, you get much more value for the money spent.
I'm just curious what the cost is.
Hours needed for prompt + Hours needed to check result + API costs.
You don't say "well let's add together the total yearly compensation of all the engineers and mathematicians at OpenAI that were involved" and throw that into the total cost. That's simply nonsense accounting.
The actual comparison you are making is some university researcher weighing between getting a grad student (several tens of thousands of dollars) vs typing up a prompt and sending a request to OpenAI for inference (as mentioned in the article, around $2000 in API and maybe a few hours for the prompt and harness).
https://x.com/polynoamial/status/2083470822258467194
Now I understand that it’s mostly the super stars benefitting from the increased attention. Folks who are less established don’t share in that glory. But on the other hand it seems like an exciting time to go even deeper for in various specialties of math by deciding where to focus these powerful tools. For every conjecture defeated some seven or eight new ideas open up. Our path through that combination will be set by creative and curious human mathematicians.
[edit: deleted a distracting comparison to Chess]
Which is less interesting work. And you probably need to do the hard grunt work by hand first to develop the skills and intuition to be able to verify an AI-generated result. So you can’t outsource everything to AI without loss of skill.
Was this different before chess computers were invented?
Thinking of this a little bit with the perspective of every new proof as a burden, dumped for review by actual mathematicians.
If you translate that to math, then all you get is math competitions, not math as a career. Of course the translation isn't nearly exact ... there's a lot more room for professional mathematicians because the math space is far more vast than the chess space and can't generally be cranked out mechanically (we have proof).
P.S. The response is nonsense ... I explained exactly why it's awful (others have too) and the response doesn't in any way refute the explanation ... rather it offers up a ridiculous strawman.
The most disappointed folks will simply drop out, but the enthusiastic ones will keep going and with luck make up for the ones who decided to quit. Chess and go certainly went this way.
However, I was looking at the proofs and reason explanation and openAI should be more explicit in how the work has flown. I find the models have jumped hoops in some places of the proofs, that can be hard to track. In fact, when a paper is published you usually get a review and if no reviewer understands they ask you to further explain the thought process. It will be fun to see if this happens here.
Edit: Oh, are you suggesting they just use it to privately improve their models? I imagine a few more correct proofs would have a very marginal benefit, if any. Also, they'll probably just get extracted, meaning it still gets out but OpenAI doesn't get to fancily announce it themselves.
[1] https://x.com/henryquantum/status/2083623695436623915?s=20
Which is very suggestive - if after everything they are not fully loaded then the next gazillion data centres being built look unlikely to be needed.
If anything it proves more data centres are needed. That's literally the only reasonable conclusion from this news.
Are you arguing there is not an AI bubble, and that all the DC buildout is fine, going to be profitable etc?
I am not looking for a online slanging match - just looking for a different point of view
I’m not participating in the slinging match but it’s very very weird that you think it’s some established thing that these companies won’t make profit. A lot of hubris must go in this kind of thought. Like.. do you all think everyone’s playing musical chairs?
Mathematicians will tear it to pieces if any of it is fake!
AI development is likely to be more compute bottlenecked than solving math problems since validation of any algorithmic improvement would likely require significant compute. But you could imagine that at this point it could be economical for a frontier lab to task 10,000 agents to work non-stop on finding novel algorithmic improvements then validating the top 50 out of 1,000 candidates on a GPT-2 sized network.
I would suggest RSI is now very close. The singularity could be less than 6 months away. I'm not saying I'd put a high probability on that, but I'd give it at least 20%, and I'd double that if looking 12 months out.
I know I'm just a crazy man shouting at the clouds, but please take to the consequences of this seriously. I understand that for whatever reason AI risk seems abstract and doesn't seem real, but this should terrify any person thinking logically about where this could all be heading.
We haven't even solved the most basic AI safety problems yet. RSI right now would almost certainly result in an extremely bad outcome for humanity.
I’m not sure it will be FOOM, maybe it will require AIs to iterate on hardware to get orders of magnitude more compute/storage/energy which would more likely require months/years, but algorithmic progress would likely saturate quickly. I guess it depends on how much you think further AI progress depends on hardware vs. software.
This was not a problem that was for sale
I'm sure they must do some of this type of work, right?
I agree, some of the problems are more difficult. I don't think that's the case for all of them. And, besides, these companies could be demonstrating how to approach problems and where their users could spend tokens to help with these problems.
Should not these companies try to work on these problems _because_ they are difficult?
We already know how to solve all of these issues. What we lack is collective political will.
And yeah, the lack of collective political will sucks. It would be naĂŻve, however, to think that there is no value in ensuring longevity in our current and future infrastructure. And improving it to sustain the population giving these companies their value is an obvious win.
Little sacrifice actually
> Solving the issue would be doing it without sacrifice
So... you are expecting magic?
LLMs cannot create resources out of thin air.
And yet this is the exact same company that has screwed up their android app so bad that the latex N^3 rendering problem makes it so having it explain it to me crashes the app.
Truly jagged beyond belief.
Sam Altman has said "If superintelligence can't discover novel physics, I don't think it's a superintelligence." Is that the test? How far away are we from AI discovering novel physics? It seems within reach.
Just maths isn't really general enough for the G in AGI.
What can we do to make conversations around these incredibly exciting and important topics more constructive? HN is where I expect to read expert comments on these topics, has this style of conversation moved elsewhere?
[0]: https://news.ycombinator.com/item?id=49157930#49132926
None of this is anything out of the ordinary; this kind of thing has always happened. The only real story here is that moderators sleep sometimes.
I for one am really fascinated by AI's advances in science and math and would like to talk about it somewhere without the constant flamewars...
People argue whether we are at y-5, y, or y+5, meanwhile we seem to be on a y=2^x exponential that keeps leading to crazier and crazier results. The much more interesting question to me is what will be consumed by the exponential like math seems to be, and what won’t. Writing has been much more stubborn, but I’ve noticed Fable to be quite a big step up there as well. How about politics? Will we develop new ways to let people express their own values in democracies, or will we get much better at manipulation?
And then there’s questions like, even if AI can answer increasingly complicated math questions, will we still need mathematicians to translate results to the real world, verify them, or decide where to push the frontier?
Yes. And there are many of them. I wonder what would help them come to terms with it. Seriously, people are going to be grieving over this. Loss of identity, loss of social standing, ideas of entire future lives that will now never happen. The greatest crime people may hold AI guilty of is taking away their dreams.
I think recognizing and accounting for your own personal biases is one of the requirements of the being an intellectually honest and rigorous online discourse participant.
Things could be genuinely impressive and fascinating even when directly challenge your ego and material well being.
Moreover, mister elite, you don't know why this press release was flagged.
I'm not sure why we should privilege your bitter speculation over more mundane possibilities.
It was always mainly a website for employees of an elite.
Never was.
eg. this: [flagged] A migrant surge tests Spain's open policies (economist.com) - https://news.ycombinator.com/item?id=49131860
is clearly marked as flagged.
Unlike the current submission: Ten advances in mathematics and theoretical computer science (openai.com) which isn't [flagged].
* https://news.ycombinator.com/newest
Is that possible or is everyone else too common to have those?
hah, sorry, we are plebs out here.
https://developers.openai.com/cookbook/examples/vector_datab...
How are the sales going?
It's more like they've already automated the parts of the jobs that the humans most closely thought of as the "their job"
My guess from following this stuff quite closely is that these companies are still a couple years away from fully autonomous research staff.
[1]. https://www.anthropic.com/institute/recursive-self-improveme...
I think we've now hit a point where 99.9% of the population gloss over these types of AI advancements because of human competence being insufficient
No human could have published this because it requires paradigm shifts (e. g. Section 5) in multiple mathematical domains. Mastering one of them to this degree is rare, mastering 3+ pretty much non existent for humans.
For some reason comments got moved to this one.
They can’t even drive me anywhere I want, though I suppose they’re getting there. I don’t think LLM companies should be surprised when rest of society hates them. They’re literally bringing in a dystopian WallE like society where most demand for human work is destroyed.
Stop coping and deluding yourself mate.
To begin with, whether AI is the one doing the discovering or not makes no difference. Any "pure math" person would aim to understand regardless - and would be quite glad that they have a longer paved path.
Any mathematician in academic or industry is more than likely not a "pure math" person (tainted by capitalism).
Isn't coping a good and useful mechanism?
>Any mathematician in academic or industry is more than likely not a "pure math" person (tainted by capitalism)
Ignoring that I meant pure as in non applied math, let's just make it clear: you agree that mathematicians who are against capitalism encroaching on this process should be allowed to dislike it without criticism of being pretentious?
The non-sofic group one is definitely a big deal - would have been a Fields medal if discovered by a human.
AI has no self-awareness. It's a tool. When you assemble a furniture using a screw driver, the torque force interacts with the molecular forces inside the metal and miraculously it transfers the force to the screw though a clever geometry design, communicating the force to the screw to turn it in a certain way.
Do you attribute the build to the tool? The "system's contribution" is helped by many other things all the way down to chips, datacenters and power generation. If the authorship requires attributing to a tool, then it should happen all the way down.
A slightly smarter highschooler could write these. I could write these. It's clear as day that the LLM, not the human, did the heavy lift. It'd be ridiculous to give full credit to whoever wrote the prompt.
[0]: Not one of the proofs in the linked article, but from OpenAI too.
I think you're over-estimating what a smarter highschooler could write.
A "finite loopless undirected multigraph" could have been explained to me at that age if we'd taken Discrete rather than Mechanics and Pure (and one module of Stats) in my two A-levels* in maths and further maths; but from what I saw of the Discrete module, neither:
nor: would have been something we'd have learned. But more importantly, we absolutely didn't have a feel for how much effort one needs to put into making sure the proof is right, so if one of us had been hypothetically asked to write a prompt it would've been no more than half that length, and missed most of the bullet points.* For those not from the UK: A-levels are between secondary school and university, when aged 16-18. Functionally they are university entrance qualifications: https://en.wikipedia.org/wiki/A-level_(United_Kingdom)
Provability is just going the way of computation. John Napier had to manually compute logarithm tables over decades and was recognised for his work; now that same work could be performed by a 10 year old with a calculator in an evening.
[0]: e.g. "go through wikipedia's unsolved math problem list and solve them".
> In particular, proofs for special graph classes, constructions of cycle covers with some edges covered other than twice, bounded-length or prescribed-cycle variants, reductions to another unproved conjecture, computational verification through any fixed graph size, and candidate counterexamples without a complete nonexistence certificate are insufficient.
which is infact a very important part of the prompt.
When the tool is a 3D printer, or any CNC system really, you bet I attribute a build to it.
I could also attribute the operator; there is no contradiction, it's a free choice, just like saying "I am in Berlin" does not contradict "I am in Germany".
What is your mechanistic model of self awareness that yields this conclusion?
> It's a tool
Does your model suggest that tools can't have self awareness?
A language model (or an image model or whatever) cannot even be sentient, and I think sentience is a prerequisite for awareness.
Even if we express a lot of our subjective experience with words, the language is just a symbolic representation of those experiences. The qualia themselves, even those that are quite abstract, are rooted in our physical presence and evolution.
You can't have an understanding of what hunger or physical pain feel like if you have no need for food or a sensory capacity for feeling pain. You can't understand what loneliness or pride at an achievement mean if you don't have a neural network wired to value social connection or status. We value connection because we're social animals that have needed each other for survival.
Even the more abstract of our subjective experiences are in some way rooted in our physical evolution.
I see no reason to believe that a neural network built entirely based on the symbolic level of language could have the features needed for the subjective experience itself.
AI awareness might actually be more believable if that awareness manifested itself in an entirely different way than in humans. But if we assume awareness because outputs resemble what we consider meaningful as humans, yet the neural network has had no inputs or evolution that could form the actual basis of human-like experience, I think we're seeing something that isn't actually there.
There is no objective evidence of qualia. All evidence of qualia are vocal or other expressions of belief in qualia. Perceptions clearly exist and are observable, subjective experience and qualia, not so much.
> I see no reason to believe that a neural network built entirely based on the symbolic level of language could have the features needed for the subjective experience itself.
If your objection is to models based on "symbolic level of language" which you think lack semantic understanding of, say, trees, you should ask yourself how our brain, based on physics which also lacks any semantic category for trees, can somehow develop a semantic understanding of trees. All of these appeals to differences with the brain never seem to acknowledge that fundamentally, the brain has the same explanatory gap with physics.
> But if we assume awareness because outputs resemble what we consider meaningful as humans, yet the neural network has had no inputs or evolution that could form the actual basis of human-like experience
This assumes a lot. It seems very possible to me that intelligence inherently develops a map of natural categories (natural kinds), and language naturally develops around such categorical understanding. Semantics are then fundamentally the network of associations between categories, eg. there is no fundamental difference between symbols and semantics, and the latter cam be inferred from the former, and that's exactly what LLMs do, and why the semantic maps between different languages are so similar and how they can translate between languages.
(*) Even if we hack around this and just do the usual trick of simply laundering statefulness to a higher level, in this case the context window being fed in, I fail to identify (**) a representation of its own state in these bodies of text that it'd be meticulously maintaining. I further fail to identify how it could be hidden or maintained, considering I control like half of it. The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line").
You'll sometimes catch models mixing up who's who and how many who-s there even are for example.
(**) I did wish for something hidden though, so maybe it's just concealed? The same way people can encode a lot more of their emotional and mental state than normal into text if they read and write a lot of it, I'm aware of research that suggested the same for LLMs, albeit I cannot cite it. Maybe those phrasing signatures are just alien to me and will never pop out. Either way, I'd expect researchers to stumble upon this during interpretability studies, and either they haven't, they have but it wasn't popsci adopted, or they're keeping awfully tight lipped about it. If you know of anything like this, your turn now, would be happy to learn.
I do wonder how reasonable it is to expect e.g. a single maintained identity though. Maybe it isn't?
(*) Another way to hack around this of course is to just precompute some internal "self-awareness states" and hop around between them. Probably the closest to what the models are actually "doing".
> a hidden representation of self that is continually tended to
This sounds like a personality? They act like they have one of those. It may be an illusion, and even if it isn't an illusion it is unlikely to be anything like the source (us), but they act like it.
> I further fail to identify how it could be hidden or maintained, considering I control like half of it.
Indeed you control everything about a local model, and much of the context of even a remote model. But the state of activations and circuits in SotA AI is hidden in similar ways to those of synapses in your head: difficult to decipher even with probes monitoring the signals directly, and often not emitted at the normal output.
> The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line").
While we can be confident that LLMs make up personas etc., it is insufficient to go from "that doesn't necessarily represent the model's internal state" to "therefore it doesn't have one".
> You'll sometimes catch models mixing up who's who and how many who-s there even are for example.
I've, unfortunately, also experienced this with humans. Perhaps they were losing their self-awareness at the time? I do wonder if old-age dementia does that by the end, though the person in question didn't ever get diagnosed with that.
> If you know of anything like this, your turn now, would be happy to learn.
Do you mean like these, or something else?
• https://researchportal.hkust.edu.hk/en/publications/decoding...
• https://aclanthology.org/2026.eacl-long.165/
• https://transformer-circuits.pub/2026/emotions/index.html
I don't see why an LLM could not have a sense of identity or personality while it's evaluating a specific prompt, or even change self awareness while evaluating a prompt since many outputs model a back and forth conversation. My point is that without a mechanistic model of what "self awareness" means, we have no way of truly evaluating such questions, we're just hand waving vague intuitions about what it could mean.
https://cr.yp.to/proofs.html
Who is going to wade through this?
It is not peer review if it is all in one company that wants an IPO.
Without a searchable index of training data, it is hard to put faith into these claims.