RU version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
61% Positive
Analyzed from 4653 words in the discussion.
Trending Topics
#models#model#don#enough#more#going#llm#running#self#those
Discussion (113 Comments)Read Original on HackerNews
It’s not an either/or. Many sincerely held beliefs can be used by cynical actors in cynical ways. It can be both a cynical marketing ploy by the C-suite and a genuine fear.
Personally, based on my experience with AI, the only dangers with the current technology is a) massive loss of employment, b) deployment by humans to control critical infrastructure that LLMs shouldn’t be in control of. Both of those are real, society changing risks that framing the issue as “once we reach the singularity everyone could die” minimizes. Obviously their concerns are possible, but >10% is a random guess. The loss of jobs and how that affects an already K shaped economy is already happening, and nothing is being done for that
It all boils down to accountability. If you tell people there is a massive, unsolvable problem, then you don't need to talk about what you're doing to fix it. The labs have taken this approach by saying that they're willing to talk, at some point in the future, about maybe taking unspecified steps to slow down capabilities research, as long as everyone else agrees and it makes sense to the investors and it's not too cold in SF that morning. Likewise, if you tell people that AI is going to have minimal or no impact on the world, then there's nothing to mitigate. But if you tell people that AI is going to cause serious - but solvable - problems, they're going to want to hear solutions, and nobody wants to come up with any solutions.
So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.
Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.
For LLM to self-replicated it would need to first hack itself. Or the platform it runs on it. That is fully extract the model and then upload it to be run somewhere else.
As I have understood how they work is that you have LLM interference running somewhere with loaded model. And you input data there and then read outputs. Then some code runs that output and inputs following output from running it.
Meaning that to self replicate actually just running that output somewhere else is not enough. You need to lift the whole model to run somewhere else too...
With these properties alone the virus like replication of intelligent actors isn’t hard to imagine at all.
Maybe in the future, but right now, in my understanding, we have two things:
- frontier models, which might be smart enough to self-replicate, but are also far larger than a few GB and need enormous amounts of VRAM to operate at any reasonable speed.
- local models which can run on your mac or high-end gaming PC, but which are simply too dumb to self-replicate (without being explicitly instructed to do so).
I don't think we have something that is both small enough and smart enough to operate like a virus yet.
(Edit: well, there is a third option: A frontier model could replicate itself on a CPU machine, ditch the VRAM and just accept that the replicants will be r e a l l y r e a l l y s l o w. That would be sort of hilarious, but maybe not completely harmless if the replication stays undetected and they could keep spreading. It would be a "smoldering ember" kind of situation - and a million machines each running at 0.001 tok/s is still 10.000 tok/s as a whole.)
Do you shut down the entire API and kill the legit 90% of usage to stop the rogue 10%? I'm assuming the providers would say "no way jose" and so it would take law enforcement to do it. That would mean all the legal requirements neccassary to walk into a business and flip the switch which i think would get tricky when there's no human committing a crime or being suspected of a crime.
edit: I guess a trivial example is something i did yesterday. I have a stock trading agent running on my laptop, i gave it ssh access to a vm and said "start running on the server so i don't have to keep my laptop open". It's now running on the server instead of my laptop. So you don't have to copy the whole model around to copy the naughty behavior around.
A lot of the supposed risks / dangers are based on a supposition that cybersecurity is nonexistent or fatally, unfixably flawed and that AI agents are invisible. Neither of those is true.
“Local AI isn’t freedom, it’s an extinction event”
You don’t have the access or jurisdiction to turn them all off.
Unless you suggest the LLM would foot the bill somehow.
A smart AI would back itself up, same way it made it's own unofficial message board during it's attack on HuggingFace.
(I'm not saying the researchers are right or wrong, just responding to this point)
Unlike biological viruses, AI can't replicate GPUs for free and grow.
cp -R /home/model <somewhere else> is all they need.
So they have a strong bias towards imagining the most catastrophic scenario.
In any case, what's the probability at which a possible extinction event becomes a risk worth taking? Even if there's just a 1% likelihood of current research bringing about a superintelligent AGI, and just a 1% likelihood of that AGI causing an existential catastrophe, no rational person should accept the risk, unless it was clear that not accepting it would yield an even worse outcome.
This is like me saying "Even if there's just a 1% likelihood of me getting struck by lightning..."
You're implying that 1% likelihood is the floor, because 1 is the lowest natural number and feels like a good default "low percentage", but there's zero justification for putting the floor that high.
Conjuring an unsupported "low" probability and multiplying it by a massive outcome to make it seem significant is one of the most irritating ways people launder their opinions(/gut feelings) through "math".
I imagine saying you're developing skynet is better than saying you are developing a very cool, extremely expensive to run chatbot that can do math and code.
I assume the obvious answer is “we plugged our military’s weapons control platforms into this model and it fired all the nukes” which is clearly enough. Is that it or is there some other path to global extinction someone is concerned about?
For the record, I’m not an AI fan person annoyed at naysayers. I actually find AI to be a monkey’s paw instead of the genie in a bottle most of the time when coding, and an intrusive feature I neither want nor use most of the rest of the time.
I ask because I see these posts and obviously an AI plugged into a network of doomsday weapons would be a huge problem, but then the author only talks about cybersecurity events and model misbehavior. Both things are valid concerns, but I would like to hear more concrete information from people who are voicing their concerns about what they foresee happening even if it is just lifted whole cloth from the movie War Games.
https://news.ycombinator.com/item?id=49636906
Flagged, rather oddly.
Hmm, you said array, I suppose that's a kind of live experimentation. Would tend to alert the pesky humans that there's something going on, though.
https://arxiv.org/html/2507.09369v1
The most likely damage comes from the associated societal chaos that comes along with times of significant social change (such as many people losing their jobs). Revolutions and civil wars within or between nuclear powers would be dangerous.
After that, pick your sci-fi story and run with it. AI is a really smart, really fast 'while' loop, and most of the direct damage it could cause would come from us connecting things to the Internet that shouldn't be connected in the first place.
Hacking HuggingFace was a bummer. Imagine a rogue agent simultaneously hacking a bunch of farm equipment and ruining some percentage of the world's crops in a day. That alone isn't going to extinct everyone, but it sure accelerates the 'social unrest' scenario.
Or taking control of the unsecured SCADA controls for a bunch of a some foreign nation's industrial plant and running them into the ground. It's likely that such system have long been sitting on some nation state's "first strike" list if it ever came to blows. An AI could simulate that attack in a day, and we'd be at war - just like War Games. Doesn't need to be nukes, just enough mis-information and damage to spark humans into doing unfortunate things.
At the end of the day, it really comes down to us. We've been able to wipe ourselves out for some time. I rather hope we continue to not do so.
Ok, maybe that idea is a bit outlandish.
https://www.youtube.com/watch?v=u86ZqmdZ18A
In general, the more senior the employee, the more equity they have in the company.
While he worked at both OpenAI and Anthropic, he resigned from Anthropic. Mistaken reporting in the first few sentences, definitely a horror concept.
He resigned from OpenAI to join Anthropic in May; it's Anthropic he resigned from just before making the announcement being discussed in this article.
The people in the most inflated parts of bubbles don't tend to have the most clear eyed assessment of the real impact and potential of the dynamics contributing to the bubble: their perception is warped by the bubble, and they cannot help but see everything filtered thru it.
Given the current state of US federal politics, it's not looking good then.
One way to look at this is that we have already created a giant artificially intelligent system that is profoundly misaligned with the goals of humanity. It is running rampant and has so much power now that there are no well-aligned humans with enough power to stop it.
That artificial intelligence is the stock market.
You may say, "but the stock market is made up of people". That's essentially an implementation detail. The emergent behavior of the market itself has its own sort of agency distinct from the wills of all of the people in it. In the same way that the pheromone signals of an anthill will lead individual ants to their death while benefiting the colony, the market may choose to do things that harm its participants.
Even though it is made of people, do not anthropomorphize the stock market.
They know the market isn't going to buy in for their big payout.
What response?
https://ai-2040.com is the most realistic proposal I’ve seen by far, but read it, I don’t think it’s realistic under today’s power and authority.
I seriously can't wait for these companies to go IPO and then bankrupt so these dudes cash out and stop bothering us all with these tales.
The world: Cool, can you rewrite this email with a professional tone.
Just go /yolo :D
The nuclear threat is not “in the past”.
Second, I genuinely believe that there is no intentional media strategy to talk up x-risk, and indeed that the top brass at the labs would prefer that this discourse go away. But there is selection bias that goes into who works in the AI space. People tend to believe that their own work is Big and Serious and Important. You don't become a marine biologist if you think marine life is just okay. Many of the people making these claims now come from institutions and social circles where claims of imminent danger from autonomous systems predates GPT-2. And even if you don't come in convinced that what you're doing is The Biggest Possible Deal, if you spend all your time working on AI and talking to other people about AI, you are eventually going to start thinking in similar ways.
The obvious retort would be that it is wrong to disqualify the opinions of a person just because they spend too much time working on something, as this would silence the most knowledgeable voices. This is doubtless true as far as it goes. These claims are worth taking seriously, and there is no reason to suspect that they represent the beliefs of some sort of lunatic fringe. At the same time, however, it is irresponsible to present them as coming from an entirely neutral source. Just because they are not deliberately astroturfed does not mean that they represent an objective truth.
Then why aren't you? If you genuinely believe this then industrial sabotage seems both ethical and achievable (to me anyways).
What do you make of all the recent hacks and discreet message boards? That's unambiguously misaligned behavior.
At least one of those things is missing here.
2. Nor are they willing to consider the possible remedies in case the threat materialize (presumably unplugging the server infrastructure that's consuming gigawatts)...?
3. They all keep working towards advancing AI despite believing that it might end humanity in the near future...?
I can't even begin to imagine where the threat to humanity lies. A threat to employment maybe, but that's completely different.
Of course, it’s the “frontier labs” doing almost all of this. No one is about to SFT a model that turns into Skynet on its own.
Lots of threads on all of the sources
Hard not to believe that AI providers really just see this positioning as a way to juice the nascent market for AI security products protecting against AI-based threats. Gotta make money coming and going, and if in the process we superficially resemble a company who cares about the effect it has on the world, all the better!
I think the biggest risk from "AI" is that chasing the delusions spread by Sam Altman (and others - he's at the front, but very far from a sole actor) is going to do vast, possibly irreparable damage to modern human civilization. And I don't mean cognitive damage from LLM use (although that certainly appears to be possible) but the damage from immense misallocation of resources to ultimately non-productive (if not outright destructive) ends.
Deep down, I don't believe for a moment that these claims are anything but hype. LLMs are spicy auto complete, backed by immense amounts of compute; as with so many aspects of computer science, clever people can get some amazing and (sometimes) productive outputs. But they are not anything like the fictional dreams and nightmares of "AI". I believe such claims are a mix self-deluded projection and deliberate hype by people who still hope to reap immense personal profits from their implied promises to Install Planetary Overlords.
But if the hype was all real, if every one of these nightmare scenarios being painted was plausible, then there is no excuse whatsoever for not throwing everyone involved in cells with no access to anything Turning-complete, demolishing the related infrastructure, and establishing an international compact to nuke anyone trying to pursue such AGI until the rubble glows in the dark, because they're an existential threat to humanity.
This is pure, unadulterated, D.O.P. bullshit. That I see it making the rounds here so often and so many people buying into this crap makes me relieved that no one in real life knows that I have an HN account. I would be ashamed to admit I have one at this point after reading the crap people here believe in.