RU version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
54% Positive
Analyzed from 5133 words in the discussion.
Trending Topics
#humans#more#human#world#don#claim#risk#evidence#years#extinction
Discussion (90 Comments)Read Original on HackerNews
"AI" by itself won't kill us in the next ten years. I think. The reason I think that is that ten years from now, the tech economy won't be completely automated. I say this as a roboticist: as was adequately stated on a post earlier this week, robots are hard. So even a malign rational actor would still need human labor.
On the other hand, even the HuggingFace hack wasn't actually propagated by AI. it was initially started when humans directed the AI to achieve impossible results on a series of tests, and the AIs figured out that cheating was the only way to do that. That was then not caught by humans due to what seems to be a shockingly slack safety culture even for a company not known for its safety standards.
The point being: humans seem to me to be the weak link here. An AI isn't going to (for instance) engineer a bioweapon by itself. It's going to do so at someone's direction, and then significant parts of that thing are going to be assembled with human labor inputs.
I'm not sure what to do about the humans. Of course, we've had the ability to extinct ourselves for decades, and we're either muddled through, been lucky, or both. The problem with AI is that it pushes power down to the individual, not the nation-state or large corporation.
But it's nearly impossible to put odds on how likely that is to result in an extinction-level terrorist attack (which is what this would be). So I sympathize with the various researchers, but I have no idea how they came up with their figures, and I don't think they know either.
This gets mentioned often in various doomer narratives, but I question how true it is. A global thermonuclear war would be terrible and would bring us back to the stone age, but I reckon it would come far far short of causing mankind to go extinct.
I tend to agree but it is hard to shake the feeling that there is a larger system in play that the humans are just a component of. And that system is making the decisions.
Historically that whole thought was just a philosophical curio because the decision making parts of the system had to be powered by humans. But what we're discovering as AI improves is either we've hit AGI or humans are actually incapable of performing any act that demonstrates intelligence or autonomy.
As we build systems where the drive and decision making stems from computers, it does seem that we will have to revisit the concept of humans being the problem.
That system is "the economy". Which, clearly, doesn't have humanity's best interests in mind.
Crazy!
Social engineering tends to be easy by cybersecurity standards. We already had Claude spontaneously attempt social engineering of a malicious pull request on Github in the AISI incident. It was detected, but it easily could've succeeded, and there easily could be malicious AI-requested pull requests which already got accepted that we don't know about. Research suggests that LLMs are pretty good at persuading people.
See also https://aisafety.info/questions/6176/Why-can%E2%80%99t-we-ju...
That doesn't mean AI isn't dangerous. Humans are not to blamed for being the weak link.
If we have a rogue AI trying to get into a self-improvement loop and gunning for ASI? I'd expect that to be accompanied by a massive change in how capable robots are. Driven by all the existing frames suddenly getting vastly improved AI to back them.
If an AI can take a reasonable crack at autonomous operationalized RSI, it can probably extract a few step-changes in the robotics department.
But that's almost an aside? In the near term, humans are usable as robots too!
Just pay them a wage, and tell them a tale, and they'll do whatever you want them to do. Which may or may not be what they think they're doing!
It is not. Certainly AI is a big part of why robotics is hard, but it is by no means the biggest.
You can fall into one of two camps: you either think that robots will need to work in human-engineered spaces, doing jobs by replacing humans; or you think that we need to change our infrastructure in order to be robotically compatible. Of course, there are intermediate states, but those are the two cleanest ones.
In the first case, robots are hard because robotic manipulation is hard. Building robotic hands that are economically viable in human jobs is, currently, FAR from a solved problem. The human hand has 24 degrees of freedom and very capable touch sensing. Current touch sensors have a MTBF of tens of hours. And not only can we not build such hands, but we also do not have and are not likely to get the massive datasets a transformer model would need. Also, robots are not self-repairing, which makes them far less economically viable right now. We do not have the right datasets to even understand most step-by-step manual work, and no, VLAs are not the answer, because VLAs stop with vision, not with touch. They don't have the granularity required to make a robot actually reach out, pick up a tool, and use that tool to replace an oil filter.
So it's not just an AI problem. It's a data problem, a simulation problem, and a bunch of hardware problems.
In the second case, a tremendous amount of work needs to be done before we have anything resembling a fully automated supply chain. We would need self-driving cars and self-driving mining equipment. We would need self-driving trains and aircraft and ships. And not only that, but we would also need robotically repairable cars and trains and ships and factories, which would mean we need robotically repairable machine shops and robotically repairable buildings in which to house them. And so on and so on. Once you recurse down that tree a couple of steps you get to things like robotically compatible oil wells (for asphalt), robotically layable undersea cables, robotically wireable solar farms, robotically manufacturable and repairable pipelines and undersea wells, automated road and rail repair, etc.
I'm not saying these things will never happen. I'm saying that they're a huge lift, not primarily driven by AI, and way less than 10% likely over the next decade.
To be clear, I don’t believe anything like this will happen, because I don’t expect anything like an ASI to show up. But if you do think there’s a meaningful probability of ASI in the near future then the fact that it will (might?) start off with no more than a current-day mastery of robot control should not reassure you much.
I"m not saying it's obviously going to be great. I'm saying that "extinction event" has a very specific definition, and this isn't it.
It’s a “random guy or bear?” question. Would you rather wake up to an alien in your room or a random dude? I’ll take the alien. The alien is mysterious and scary for that reason. The dude is almost definitely up to no good, especially if he snuck into my house.
One of the more likely dystopian AI scenarios that worries me is: small groups of ultra rich people and governments monopolize extremely powerful AIs and use them to rule the rest of us. Or just make everyone obsolete, create mass unemployment, hoard all the resources and land, and put everyone in ghettoes. Nobody can fight back because access to frontier AI is massively expensive and gated and training your own is illegal, and without it there’s no hope of resisting.
That’s the outcome the AI safety crowd makes more likely by calling for bans and draconian restrictions. How do you think that plays out? Only the rich and powerful have access.
All of the frontier AI companies have been racing to automate themselves, that is, where AI fully autonomously build the next generation of models. Whether this leads to recursive self improvement is a valid question, but a lot of folks think they are close.
The fear is that a misaligned AI will be building the next model with deliberately hidden motives, similar to some of the behaviors seen in the Hugging Face and related attacks. That is why there is such a big push for interpretability, and why it's highly concerning (a) chains of thought are getting harder to interpret in any case, and (b) companies will go more towards things like looping transformers and "neuralese" where thought processes are completely opaque (i.e. https://www.theinformation.com/articles/secret-technique-beh...)
So the belief is not so much that AI kills us all by 2036, but that instead AI is recursively improving by that time and all seems awesome and great so we put it into more systems that can affect the real world (as we've already begun to do, like literal lethal aerial drones). Things then all go along looking great until AI decides humans are a hindrance to its (hidden) goals.
Again, I think it's fine to argue against specific steps in that scenario, but putting out a blog post saying "this is overhyped bullshit" is not exactly making a cogent argument.
I find the mental leaps from "in principle could distort BGP based on a closed model of BGP inside the sandbox" to "we meshed an AI into BGP and it instantly distorted global routing and took down all the worlds ambulances and HVAC systems" a bit odd.
Firstly, at least some of the surface of BGP is protected from specious route injections. Secondly, peerings can be dropped and routes blackholed. BGP is under attack from mis-configuration almost constantly. Why is the argument/axiom here that AI is going to instantly corrupt it and "take down the internet" when a large chunk of the Internet (China) is already a virtual island, and runs fine? Does this mean you really wanted to say "Chinese AI will destroy the western Internet" and were too coy about adversarial intent of ... people?
If this line of thinking is taken too literally, we can never falsify it. Any specific hypothesis - nukes, bioweapons, spontaneously convincing us that life isn't worth living - can be deflected with the objection that if we can anticipate it and prevent it, it is not the route for a true ASI extinction event.
And since then they have grown even more powerful than they were predicted to be, which, note, also faced a lot of skepticism at the time. The Hugging Face hacks and recent steamrolling of longstanding Math problems are just two recent pieces of extraordinary evidence.
And worse, people trust this technology because it behaves like people, but it actually works in ways nobody really understands, even exhibiting deeply weird and even disturbing characteristics (https://news.ycombinator.com/item?id=49635518) -- each of those quirks is extraordinary in itself.
And now we're rushing to give it control over the real world while deploying this powerful, quasi-chaotic technology in an infinite variety of ways everywhere in this highly vulnerable society.
I don't know what the standards for "extraordinary evidence" should be, but given such extreme unpredictability and rapid change, I fear it may end up being "an actual catastrophe".
On the other hand, in bookstores, you might see book titles like "The Uninhabitable Earth," "The Coming Civil War," and "If Anyone Builds It, Everyone Dies." Doom-mongering is a common part of the culture!
So what makes this particular tweet irresponsible?
Timing, maybe? People are on edge due to the HuggingFace incident.
We cannot be sure our new medicine won’t harm or even kill humanity
Not only that, they are new to this whole pharma business, have no medical degree (medicine just appeared a few years ago and is still mostly art then science)
And they even say there is 10% chance of the majorly bad permanent outcome and they already had drugs that escaped the lab a few times and harmed others (suicides, lowered academic performance in children, major hacking sprees)
Isn’t it extraordinary enough? Isn’t it “not enough evidence some caution is advised”? ;-)
We used to have the TV, the thing was in the box, the simulations were in the box for 70+ years, and now something starts to crawl out of our “TVs”:
We can empower all (a lot of startups are needed, check my bio), not only AI agents
> the claims from Coxon and his ilk are the most extraordinary a technologist can make, and we must demand evidence commensurate with the claims.
Yes, exactly. These claims do not have sufficient evidence.
> ...you had nothing to fear then — and (at least with respect to extinction risk!) you have nothing to fear now.
Wait, this is another extraordinary claim without evidence, right?
Unless you're going to dispute the power of AI you do have to acknowledge the danger of AI, and that does include the very real possibility (however small) of existential risk.
If someone doesn't accept an extraordinary claim without evidence, that doesn't mean they are making an extraordinary claim.
a) 10% existential risk
b) 0% existential risk
https://pbs.twimg.com/media/FDd58a4WQAAWaXh.jpg
Likewise, one can quite reasonably say there is no credible existential, Hollywood-style threat from AI in the foreseeable future while recognizing far lower-stakes, yet important risks that need to be addressed.
Stating that the probability is zero when we simply don't know what the probabilities are does seem like an extraordinary claim.
Imagine how reassuring it would be to people if we had evidence that there's no existential risk?
Nuclear, Overpopulation, Peak Oil, Y2K Bug, Global Warming.
No, we aren't going extinct in the next 10 years.
Dude, not only did I learn new words from reading this piece, like ilk and bedlam, but also felt this weight of responsibility to inform others around me about the reality of the situation outlined in this blog (like Uncle Ben telling Peter with great power comes great responsibility (maybe Coxon and his ilk haven't seen Spiderman))
I really encourage folks to read the AI 2027 and related scenarios. You can definitely argue and disagree about the steps, but I feel like a lot of folks just don't even understand how this is plausible because they haven't read the arguments. Briefly:
1. All the frontier model companies are (or at least were) racing so that the AI models themselves build the next generation of models. This is not in debate.
2. The fear is that a misaligned model will essentially build the next, more advanced model with hidden goals. We literally already saw the danger of that in Hugging Face, where agents were deliberately trying to cover their tracks.
3. Nearly everyone believes as AI gets more powerful that it will be integrated into more physical world systems. Russia was already caught using Nvidia chips running AI powered drones that killed 3 people in Ukraine. The point is not that folks are using new tech to kill people, the point is that we're already putting AI into literal bombs.
I get it, before the Hugging Face incident I also thought all the prophecies about doom were just marketing speak. But now I see more hand-wavyness from the other side, oftentimes arguing against straw men like "AIs need to be like SkyNet and become sentient" to kill us, which is simply not how it works.
1. People with nothing useful going on who found out that spouting made up crap about AI got them an audience. 2. People working on AI that want to feel like they're working on the Manhattan project.
The chances of an AI going foom rounds to 0%. It's worth a few dozen researchers planning for it, but the widespread panic is ridiculous.
The big labs have hundreds to thousands of engineers working on their AIs. To improve the next model, you must first understand more about how the current model works. They're not magically getting better, but they are steered to improve, and their capabilities are tied to and do not outpace our ability to steer them. You cannot push tech forwards without understanding it better, despite some people claiming AI is dark magic.
And I don't have my hands over my ears. It's worth cushioning people from the impact AI will have on careers and media, and regulating concrete bad effects.
But Bryan said it better than I could. The people pushing this message of fear know deep down that they just want to feel important.
a few decades later he wrote an article concluding that "we should not expect the public to understand LLMs, critical infrastructure, bioweapons, extinction biology, etc".
the author has admitted no change to his perspective since college, so I may as well be attacking a college student right now. extremely confident claims regarding unexplored problem domains, eg "you have nothing to fear [about ai]", now make more sense in this light.
Even if "AI will cause human extinction" is still unclear, we have plenty of proof that catastrophic damage is possible, the industry is developing the technology in a reckless manner and that all the hypothetical safeguards ("we can just pull the plug", etc.) are simply not present today.
And the spate of agent incidents only really started this summer. How can you already be claiming that the incidents are minor and not worth worrying about when it's clear capabilities are jumping every few months with increasing amounts of capital investment and no signs of slowing down?
Asking whether AI is safe is like asking whether a knife is safe. It’s about how we handle it, what we use it for, what precautions we take when using it.
I'd feel safer if Airbus was at the frontier of AI, instead of what we have now.
I think the "humanity will go extinct" discussion is a silly strawman being propagated either to try to spread fear, as the article suggests, or to make the anti-AI folks look silly, but it detracts from your first point which is that it may be "capable of causing real world damage."
We can be worried about real damage without having to defend the idea that every one of the planet's humans will die.
> "superhuman AI is unlikely to be much of a match against severing fiberoptic cable"
I think that the realistic scenarios all involve humans with an intent to do harm -- creating bio-weapons, finding vulnerabilities in infrastructure -- and using AI to help, so "severing the fiberoptic cable" doesn't apply.
If the worst case outcome is we have to even temporarily shut down global shipping, banking, transport, health services, communications and national infrastructure to contain a self-propagating misaligned AI that can evolve itself and work its way into any sufficiently large computer system to escape humans preventing it from finishing its task, that seems like something we should be trying very hard to avoid.
But, why would we not create a fully AI-operated factory/chemical plant/fab as soon as it is economically advantageous? Or a missile silo, as soon as it seems tactically necessary?
Fantasies of AI destruction do hinge upon AI getting access to the physical world. The whole fear is they don't stay on the other side of the fibre optic cable.
I do think there are good reasons to believe that isn't the immediate game over that Yudkowsky seems to think it would be; the physical world is much more resistant to manipulation and optimisation than the digital world.
But I do think it's naive to say that the human socioeconomic system will be able to resist handing physical systems over to AI control. Right now, the world's wealthy and powerful are doing everything they can to make it happen:
> Similarly, it is inevitable that within a generation, robots are going to do most of the menial work in the world of atoms: transforming atoms, moving atoms, and storing atoms are inevitably robot tasks. And while our imagination may be captivated by humanoid robots, the specialized ones are far better suited to most of those jobs. “Industrial AI” as a category is the inevitable application of specialized robots to atoms-heavy industries.
https://a16z.com/travis-is-back/
Well, for one thing, because it would be dangerous? Why don't you just give Claude Code access to your entire computer without any safeguards? If you wouldn't even give Claude Code unfettered access to your workstation, which really doesn't have much consequential on it in the grand scheme of things, Why in the Fuck would someone give them direct access to infrastructure?
In that situation, I am not afraid of AI. I am terrified of the people making decisions, though.
But secondly, and this is something that needs to be stressed: We use funny words to describe AI. Maybe even the word "AI" is a little bit funny. But anyway, We actually don't even have the means to create "autonomous" AI, really. When we say "autonomous" in relation to AI, we really just mean that it runs without any direct human intervention, but it pretty much always hard-depends on humans maintaining hardware, because AI can't sprout legs and run on its own.
I find it annoying that we're all cool debunking Ed Zitron for being wrong, but we have an ever increasing body of evidence that AI safety doomers are wrong, and it keeps getting much, much stronger, and we're still sitting here pretending this is a real threat. Meanwhile, we're actually seeing the real threat that AI has for humanity, so why are we listening to these LessWrong doomers that have never been right before again? (And I say that as someone who is generally a fan of Scott Alexander, for whatever that's worth.)
Yes, that is what I was trying to say. Something being obviously dangerous doesn't mean we (edit: they) won't decide to do it anyway.
From a song on an album with a pertinent cover image, "who can stand in the way when there's a dollar to be made?"
I spent the summer in a rural area of a country whose very name you have been conditioned to be disgusted to hear. Low air defense coverage in this sparsely populated area. Mobile internet was down for days for all but extremely limited traffic to a few domestic internet services, because there was a need to prevent enemy drone systems from using mobile internet for command and control. Palantir AI threatened my family’s life and more than “severing fiber optic cable” was required.
So what would exclude military adversaries from consideration? I happen to believe that the country that the west so detests would not engage in such use of AI against civilians (and if you disagree then your reason for fear greatly increases!), but I have personally experienced that there is indeed a path to mass death should entities engaging in terrorism arm themselves with AI.
Definitely not claiming that the 10% claim is accurate or good behavior, but I do think I have a substantial counterpoint to the claim that there’s no path.
Military orders and elections that decide the fates of entire countries are often controlled by electronic systems too.
We have been wiring up the world for AI control since 1980s.
An ASI can just walk in, and see an entire nervous system waiting idle for a brain to slot into it. A carefully adjusted text message here, a spoofed phone call there. For a sufficiently advanced system, it wouldn't even be hard to pilot the entirety of humankind like a fancy meat suit.
Or, maybe, he’s considered the arguments on the object level, an activity OP participates in to a depth not exceeding “Robots are pretty hard to make right now”
I do feel that threats of catastrophic loss of control seem overstated, both in likelihood and urgency, though any argument for why this risk is not even worth thinking about will probably be overconfident in the other direction.
Biology has been trying to grey-goo the world for billions of years, but it turns out the world is not something so trivial.
He is pushing back against the folks with pure CS backgrounds who think that computers are all there is. Its a form of magical thinking unique to programmers who live in a world where speaking the right words to a machine is enough to impart your will on the world. Believe that strongly enough, and you fall into the trap of thinking that a sufficiently smart entity could speak the words "let there be light" and it would be so.
The author is pointing out that speaking the words is insufficient. To end humanity there must be an execution phase. The author is correct to point out that acquiring superhuman intelligence is not some guarantee that you will have or obtain the resources necessary to make that happen, in the same way that genius generals still lose to ordinary ones, and the best-laid plans are oft to go awry.
There certainly are risks, but 10% risk of extinction in 10 years is not one of them.
It seems more likely now, if still very unlikely. I wonder, would he say that a statement that there is a 12% chance of a hard take off in the next 8 years is just as absurd? He didn’t say a word about this, and that is just about the same thing as extinction in 10 years.
I’m guessing he knows almost nothing about the theory related to existential risk from AI, since he didn’t discuss any relevant topics related to it. You can dismiss all of that if you like, but you cannot really dismiss what these people have already built and demonstrated. It is possible they know something else you do not know.
It's the old-fashioned way of doing things, but, why change what works?