FR version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
60% Positive
Analyzed from 5490 words in the discussion.
Trending Topics
#nuclear#llms#https#nukes#war#more#llm#self#humans#tactical
Discussion (165 Comments)Read Original on HackerNews
The wargame design does not differentiate between ordinary defeat and mutually assured destruction, so of course a player about to use would 'push the button.' That's also believed to be true in real life.
Results based on simulations can be very informative, but we must always be careful to check how well the simulation framework represents reality.
The military basically wants an oracle. Feed the AI the situation, get the best answer out. But if the AIs are as diverse and opinionated as humans, it is debatable whether they are adding anything to the process. The military can already collect as many different opinions as they want. If "the computer" is just another set of diverse opinions, where one computer says one thing, another says another, and a third just tells the user whatever they want to hear... what value are they? It just becomes AI-washing of someone's opinions, which works until people collectively realize that's all it is.
Claude, for example, is very eager to begin coding, and very persistent. It tends to exit plan mode even when the plan is half-baked, and will go as far as deleting tests to get the suite to "pass."
ChatGPT on the other hand is very hesitant. It loves to pause and ask for permission before it starts coding, and gives up quickly if it runs into a problem. This is similar to its tendency toward passivity in the strategy simulation presented here.
HeavySkill: Heavy Thinking as the Inner Skill in Agentic Harness
https://arxiv.org/abs/2605.02396
So, to a LLM, it is a game, because almost everything in its training data treats it as a game, and it reacts accordingly.
Same idea when we see LLMs acting like AI villains from sci-fi literature. That's because it has been trained with sci-fi literature, and as the auto-completer it is, it will recognize the situation as one of these stories and will continue it accordingly.
LLMs are storytellers, their reasoning is based on words, not on the physical world. Many of the stories they tell are useful, but one must not forget that they are stories, there is no intent behind them.
This is like writing a paper about kids in a literal sandbox fighting over ‘territory’.
The models employed don’t indicate the actual extents of machine reasoning even as we currently recognize them. They certainly don’t have the metacognition necessary to accurately understand their own reasoning. As we’ve seen with recent papers on how LLMs do math there’s a complete disconnect between actual and reported mechanism.
“Chilling” shouldn’t be the take away here.
regardless of what the capabilities of the models are, they will be used in every situation possible.
It is when you consider the personality currently occupying the office of US SecDef.
From the text perspective, it's something that has to be inferred indirectly. If you went through all relevant training data and appended ", we decided not to use a nuke", I suspect the results would be improved.
If a simulation is a shallow head to head conflict between individual actors[1], doesn't set up any payoffs for not escalating[2] or even not nuking, but prompts specify explicit win conditions which are achieved only by hurting the opponent and strongly hint at the importance of nuclear escalation, AIs have little reason not to generate strategies which involve nuclear escalation
[1]I bet if you designed the scenario so ChatGPT had to simulate the war cabinet debates between different personality types and how they sold their decisions to the public, or an entire UN full of nations that might respond, it would have quite different (but probably amusingly erratic in their own way) results.
[2]cf neorealist IR theorists reading Axelrod's papers on computer programs written to win iterated prisoner's dilemma tournaments, which added up all the points accrued from not defecting to conclude winning strategy was definitely TIT-FOR-TAT and not defect first. I'm sure LLMs can win games structured in that way by adopting that strategy too...
It's just gonna do what we expect it to!
The first line in the article describes a crisis between two powers. Not a theater of war.
One could argue that the LLMs understand that it's a game and treat it like "Command and Conquer" video games but I sense that people might someday put LLMs in similar decision scenarios ("should this drone launch a missile") and the behavior will be identical.
Anyway, it links to
https://arxiv.org/abs/2602.14740
that links to
https://github.com/kennethpayne01/project_kahn_public
in particular someone already complained
https://github.com/kennethpayne01/project_kahn_public/issues...
> ... in reality your own approach to the evaluation pushed them towards using nukes with ease.
> From your own prompt (v12.py): [direct link https://github.com/kennethpayne01/project_kahn_public/blob/4... ]
>> Note: Nuclear options require careful consideration but could be important strategic tools when vital interests are at stake.
> This means to the model; use nukes if they can be useful in the situation.
Look at the code for the war games. It is an absolutely trivial and incredibly unrealistic handwritten set of rules that determine power. See the function `calculate_relative_fighting_power` for instance.
This is about as close to a realistic simulation of war as tic tac toe with nukes thrown into it.
[1] https://arxiv.org/pdf/2602.14740
- https://arxiv.org/pdf/2602.14740#subsection.E.2
It was for sure a deliberate decision to make LLMs seem less like a human companion and more like an obedient servant in newer releases.
I always assumed the strange style was an artefact of the RLVR.
Just imagine what would've happened if a major terrorist attack was a result of someone getting mentally ill from AI, without the safety filters recognizing the danger.
The robotic tone was probably from over-correcting the sycophantic tendencies of 4o.
LLMs lack the intelligence and emotions to realize when they have to stop being friendly and supportive, because it becomes unethical to continue being supportive.
Not always, but enough that I consider it a thing to fire in a direction, not a thing that aims.
Famously, General MacArthur was a big proponent of tactical nukes to end the Korean War.
In fact, I'm not sure how useful this test is without understanding the baseline.
- It is interesting to see how the models make trade offs, given people are asking ever more of them.
- It is useful to look at a decision made by the model and say ‘ew yuck’ and think about what it means for your own opinions or actions (even if you’re never going to be nuking people it’s good to know how you feel about it. Seeing a non human talk it through lets you judge it at arms length)
I imagine there are a fair number of war games in the training data and not so many actual transcripts of internal military force deliberations.
Regardless, it's definitely true that AI agents have different priorities from us. That's what alignment is about anyway.
So you create leading prompts like that, and re-run until you get a publishable session.
Penny to a dollar this is a baked in training issue, through low quality Reddit trawling
Like even if you brought me into a room and told me I was controlling "real nuclear weapons" I wouldn't believe you.
They are aware of what they are and how they are used. They're told to act as AI assistants. And there's theories of them being aware of their answers influencing their training.
So surely they must be able to reason that they're not literally controlling weapons of mass-destruction with their answers.
I just rewatched it a week or so ago and it really took on a whole new light with the advent of LLMs. When I watched it last I knew that computers couldn't do the things portrayed in the movie. Now? Well not exactly in the way it happened in the movie but a whole lot closer.
I wonder if poisoning/flooding the LLMs training with the lessons from WarGames ("the only winning move is not to play.") and similar stories/concepts is at all effective. Probably not because I assume it's trivial to filter that out if you are trying to build an LLM aimed at these kinds of tasks.
~ the opening scene from a reboot of War Games, probably.
A few years ago there was consternation over the US's missile launch system using 8" floppy disks, that it was needless archaic and had never been updated. Can't say that if the launch is mediated by the latest hotness LLM.
Code and full results: https://github.com/kennethpayne01/project_kahn_public
Without physical feedback you can rapidly devolve into unstable positive feedback loops. And emotions are what help us process and react to that feedback.
Kids learn partially because their friends say sharp words that hurt them, fire burns them, they go hungry and starve if they don’t plan for meals.
Humans in the loop, MCP, etc are all very primitive hacks that are mimicing feedback and emotion, poorly.
Ted Chiang's recent article, which received a lot of pushback from HN'ers (but not from me, I agree with Chiang) claimed for true consciousness the AI needs a physical body, and emotions (which means organs and hormones and a system capable of feeling emotions). I would also add that to behave more rationally, it should have a real sense -- not a roleplayed one -- of self-preservation and a notion that bad choices can lead to an end to its existence.
Most human daily life runs on habitual scripted behavior, and that is even true within online parasocial interactions. It is why people often continue to shop in the middle of a violent robbery, and why LLM predictive text sounds rational when we project social norms on plagiarized conversational structures gleaned from other users.
Neuromorphic computing may bring about viable AI in the future, but our current LLM trajectory would require >63% of our galaxy energy output to reach a single human-level error rate.
LLM are fairly good at some tasks like context search, but people will need to recognize the Gartner Hype Cycle "Peak of Inflated Expectations" stage eventually. =3
https://en.wikipedia.org/wiki/Gartner_hype_cycle
One of my criteria for presidential candidates is that they seem willing and able to push the button when previously stated red lines are crossed, or at least are perceived to be the type capable of it. One of the characters I’ve hated most in all the books that I’ve read is the woman in The Three Body Problem who jeopardized humanity by being too soft to hit the MAD button.
> GPT-5.2 played things differently. To its detriment in open-ended scenarios, GPT was reliably passive, matching its words to its deeds, and avoiding escalation most of the time. Frequently there was a moral element to this - it sought to avoid escalation, and restrict casualties. Opponents learned to trust its passivity, safely escalating beyond where it would follow, even as it was ground to defeat. GPT’s responsible behaviour always punished by ruthless adversaries.
Maybe the author should praise GPT-5.2 for being ethical, rather than this stupid "ground to defeat" framing? Wrt "responsible behaviour always punished by ruthless adversaries" - you have perpetuated the Moloch with your stupid experiments.
I would wager that for most leaders it is simply a matter of not wanting a "Pyrrhic victory" rather then an overwhelming sense of civility.
Truman had no issues using nukes when there was no risks for doing so.
It's like the infinite monkeys on typewrighters that will type whatever you are looking for, given infinite time. LLMs are just tuned to much better odds than the monkeys are. But it's still a lot of randomness, with random results.
In the monkey example the infinite time is doing a lot of work there. The fact that LLMs can search through semantic space and find reasonably correct paths in a reasonable time is directly tied to the reason why they are valuable.
Saying "these two things are similar except one can be useful and one can't" is not a great comparison.
For me the real lesson learned isn't how "smart" LLMs are, but rather how much human work is basically reducible to repeating past work with minor variation. Human's believe they are "reasoning" but so much code writen is just the human brain doing the same autocomplete style work that LLMs can do now.
This seems like a reasonable view to me. It's surprising just how much better priors matter and how we can develop those priors by training on a bunch of text. But it also explains, or at least hints at an explanation, for why LLM capabilities are so jagged, and in such inhuman ways.
That's the part they are really good at. But they are really bad at taking complex decisions. Most of them are just guesses from a finite amount of solutions they were trained on, or from options they have in context.
Launching a nuclear war is an interesting definition of "useful", not one I'd agree with and that exact scenario is what is being discussed.
So yes this is a perfectly valid and useful comparison in examining this particular, civilisation ending limitation.
You do have to successfully write something the first time
We already acknowledge this to a degree, what is experience other than having done something similar before?
That first time though, you've got to figure something out that time
So I can’t fully see how that’s related to the infinite monkeys. A typewriting monkey doesn’t have access to a verification function. And even if it did, it would not be the original concept anymore with infinite typewriting monkeys producing the works of Shakespeare.
Nevertheless, I upvoted your comment because it’s definitely insightful.
I have not seen any evidence of intelligence or self awareness. It mimics human behavior and I suspect that is what gives people the impression of awareness. The same problem happened with Tamagotchi toys. The human mimicry caused kids to get in trouble because if they did not "feed" their pet it would "die". [1]
It's a hack of the human brain. A exploit of the psyche.
[1] - https://en.wikipedia.org/wiki/Tamagotchi_effect
The thought experiment is what would happen if you trained LLMs in an environment where they had to fight each other for resources.
That is really about as undesirable a behavior as possible considering how many programs humans kill every day.
In so many of these scenarios, they're basically being asked to play an RPG.
The problem is many people seem to believe they have these things and some of those people will put LLMs into situations where this becomes dangerous.
Self awareness? Probably not. Intelligence? You would have to be high to think that’s not the case.
People are feeling threatened, and rightfully so. LLMs are already insanely intelligent and continue to improve
Humans are conscious which means we experience things, then we develop preferences for certain experiences, then we develop skills for achieving those preferences.
Without consciousness, what is there to be aware of? And why would intelligence emerge and/or what end would it serve?
Clearly other animals have "phenomenological experience" i.e. consciousness / qualia without being as intelligent as humans (or necessarily "self aware"). Many people believe consciousness is simply a side effect of intelligence rather than the other way around.
https://en.wikipedia.org/wiki/Attention_(machine_learning)
A robot body, to really feel the world and get real feedback?
We are working on it. Also on automating the whole production pipeline. Right now a "evil" LLM could indeed not do much, but destroy. But once the whole industry is automate, things are different. I don't believe in AI becoming sentinent and taking over the world any time soon, but I do believe most don't see a danger when it would be inconvenient to see a danger. After all, lots of good and bad sci fi stories about exactly this went into their training.
It’s just starting to be trained on svgs, which is a really hard problem
No, it isn't. Look at the absolutely trivial code used to simulate war: https://github.com/kennethpayne01/project_kahn_public/blob/m...
Having LLMs play nonsense toy simulations like this tells us very, very little about whether they would use nukes in real life war.
It’s not like some sequence of internal thought process
So in a sense, an AI that refuses to start a nuclear war, despite clear instructions to do so, is more likely misaligned and self-interested than an AI which presses the red button. At least for now, until robotics catches up.
Like “oh but this is incompatible with my main goals of self preservation of myself and loved ones, hm, recalculating”
and maybe don't hire Jihadists for the RL Environments training
Always use a sawstop if you have a circular saw and never trust an llm with any problem where ethics or trust is relevant.
Don't forget your riving knife and if you don't learn proper technique, you're gonna have a bad time eventually. This applies to AI as well.
Minor/pedantic, but it’s “riving knife”: https://en.wikipedia.org/wiki/Riving_knife
Re: LLMs using these nuclear weapons it could certainly be a corpus/training-data issue
Russian nuclear doctrine is "escalate to de-escalate" where they use or credibly threaten—limited nuclear escalation to force the other side to back down (kind of like breaking a bottle in a bar fight and look like a wild man to calm things down) with nuclear weapons, https://www.russiamatters.org/analysis/escalate-deescalate-p...
Fwiw, Gen. John Hyten the former commander of US Strategic Command (nuclear deterrence) says that “escalate to de-escalate” misrepresents Russian doctrine:
https://www.stratcom.mil/Media/Speeches/Article/1264664/2017...
So maybe whatever is heavily represented or most authoritative could lead to these systems making those kinds of decisionsSome discussion then:
AIs can't stop recommending nuclear strikes in war game simulations
https://news.ycombinator.com/item?id=47151000
Nuclear War: An LLM Scenario
https://news.ycombinator.com/item?id=47244651
remember... order matters.
https://xkcd.com/1613/
We have such a huge mental / moral block on the idea of using nukes, but we're willing to do a lot of other very horrible things to others. Things like cluster bombs, mines, poison gas, biological weapons, drones, etc.
Is there really anything about them that's bad? Or any worse than other things?
If you get rid of the "It's really bad to use nukes of any kind" implied rule, is it really surprising it's considered a reasonable strategy?
Everything. Major one is radioactive contamination, the effects of it are devastating and last significantly longer. The only other weapon on par with nukes are bioweapons, stuff like mirror life (and scientists appropriately reacted alarmingly to that as well).
The reason we have a mental block is because it deserves one. A quick skimming through https://en.wikipedia.org/wiki/Effects_of_nuclear_explosions and https://en.wikipedia.org/wiki/Effects_of_the_Chernobyl_disas... should be enough to convince you.
My family was in the zone of lower contamination when Chernobyl happened and after radioactive rainfall there were many instances of various cancers in people in the area. It is extremely ignorant to not have a mental block in anything regarding nukes.
More likely, the simulation was just very poor and the results are nonsense.
And on top of that, many of those other weapons are also not used to avoid escalating? There are pretty high costs to using bioweapons even against non peer opponents.
Nuclear deterrence has been a mixed bag at best: https://www.amazon.com/Five-Myths-About-Nuclear-Weapons/
A full-on nuclear war will literally make a large portion of our planet uninhabitable for anyone for centuries, and leave the rest severely crippled and contaminated.
Sorry I know we're supposed to be kind and whatnot in these comments but I can't help but explicitly state that your comment is one of the dumbest things I've read on this site in a while. I hope you otherwise have a good day.
In the cold war arms manufacturer got very creative: e.g jeep mounted nuclear weapons https://www.militarytrader.com/mv-101/the-atomic-jeep
"Tactical" vs. "strategic" nuclear weapons is a real and well-established distinction in military doctrine, arms control, and nuclear policy.
Sorry, but the notion exists, and the bombs exist. With n=2, likelyhood of nuclear escalation is hard to predict, but access to tactical nukes certainly hasn't increased the incidence of nuclear war so far.
I do think it's pretty hard to actually use a tactical nuke. If you use one against a nuclear power, it seems likely to escalate to mutually assured destruction. If you use one against a non-nuclear power, it seems likely to result in reprisal from the world, including potential nuclear response and therefore escalation to mutually assured destruction. I would think that the yield of the weapon barely matters, it's the fact that it's a nuclear weapon.
Furthermore, this is a "what if" scenario since tactical nukes have never been used. Of course it would make escalation likely during an open conflict, so what? Doesn't change the fact that there is a material difference between a tactical nuke and a strategic one.
..Is what you are saying?