HI version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
56% Positive
Analyzed from 10568 words in the discussion.
Trending Topics
#humanity#world#humans#don#more#human#feel#where#life#things
Discussion (207 Comments)Read Original on HackerNews
We will very quickly get to a point where very few people will be able to contribute economically because they will be worse than AI (including robotics) at most domains. A world where people just have their whims catered to is not a utopia. We have tons of sayings and idioms about this, e.g. "no pain, no gain", "only the hard stuff is worth doing", etc.
The humans all running around on their Wall-E carts doesn't feel like utopia to me.
Edit: to head off the "rich people worried about striving for their job while people in poor countries are starving" arguments, I encourage you to visit a dying rust belt town in the US. These folks are by and large not starving (indeed, overabundance of calories is the bigger problem), they have adequate shelter, and plenty of outdoor space. Their material needs are met. But these places are despair central. People see little opportunity for advancement or the ability to change their circumstances, and the opioid epidemic is a direct consequence of a sense of hopelessness that there is no place for motivation to go or to effect change.
I understand where you're coming from on that, but there are a ton of people living in slavery, or in unsafe working conditions, or with food insecurity, or dying of preventable disease.
There will always be something to strive for, even if it's made up. You think that the best football players in the world are doing something real? No, it's a made up game. We'll make up more games.
Regarding entertainment jobs like sports star, musician, artist, actor, etc., the problem is those are professions where, for the most part, only the very best can make a living. Nobody is that interested in seeing the matches of the 324th best tennis player in the world, despite that person being in the top .01% of tennis players globally. Similarly, only a tiny percentage of actors "make it", everyone else does it for a few years and then leaves because they enjoy eating, or they accept poverty wages for a long time.
I get pretty scared when I hear the AI utopianists rattle off all the benefits that are coming (and when they play down the risks) because it feels clear to me that they really haven't wargamed out what society would look like with very powerful AI.
As a more silly comparison, forklifts didn't stop people from weightlifting.
This reminded me of something. https://www.sbnation.com/a/17776-football
I somewhat get the point, but I think stuff like this is very often said by very privileged people. I don't think it's a serious threat to humanity as a whole. Perhaps to some humans, but there will always be ambitious people, even in a world where most things are provided by benevolent AIs for free. There is simply so much more harm caused to actual humans today by resource insufficiency than by them having too much, that I don't think we should worry about this scenario today.
Also I believe our socialness was crafted out of the need for survival. If removed it's all gonna run wild since nobody needs anybody, and it might end up as doom scrolling but for human relationship. Short bursts from time to time and that's it
In many ways they don't apply to humans, and I think a lot of the modern critiques are valid, and the overpopulation concerns in particular don't map well to the modern world. Still, I believe it helps to think deeply about what happens to societies when innate drives and motivations have "nowhere to go".
Now you might say “but those would be fake jobs.” If so, I have bad news about how many present-day jobs are fake jobs.
> There are even bioengineered human-like creatures (to humans what corgis are to wolves) sitting in office-like environments all day viewing readouts of what’s going on and excitedly approving of everything, since that satisfies some of Agent-4’s drives.
I'll leave it up to you to decide if that's an enviable/desirable future.
Bill Gates talks about this in his recent missive, but as robotic technology improves (and, perhaps more importantly, as the cost comes down) there will be huge competitive pressures for companies to replace workers with robots wherever they can.
wut.
Just look at the legions of people that have moved from Chinese rural areas to big cities. For many, many people, life in the cities is worse and kind of dystopian - much more overcrowding, "drone-like" jobs indoors, horrible pollution, and especially for men an often abysmal sex ratio if you want to find a mate. But people do it because it offers the dream of advancement and a better life - and of course some people do achieve that dream, even if the majority do not. That's versus life in rural areas which is guaranteed to be static, no chance for advancement and often crushing boredom.
People everywhere want to strive, work hard and get ahead. This is not just a privileged rich people issue.
https://web.archive.org/web/20100211181147/https://marshallb...
> superintelligent AI will inevitably destroy humanity seems to have no flaw
For a section of population who were in a certain age-range when the pandemic hit and that derailed their certain kinds of plans/hopes/dreams in personal life and then this LLM/agentic world moved in, especially if they are in an industry directly and most affected this, this has been a surreal and slow Kafkaesque nightmare for a few years. The thing is with time and age passing hope keeps getting lost inch by inch and every lost bit of hope aids in killing the next bit. Then comes the sense of dread - the unchangeable reality that age passed passing means there are certain experiences and milestones that are essentially out of reach because there's no going back of this clock.
So I feel a strange kind of relief about this possibility of doom. I am just sharing what I feel, and I don't feel good about feeling this sense of relief that after all it might end for us all and not just an unlucky few. This feeling and the calmness about it saddens me.
I think about the incredible luck I had to start my career in the .com boom, to have lots of hope about technology and to have the economically valuable skills that were desired.
I get angry when I see people my age or older complain about the apathy of young people. I don't like the apathy, but I find it trivially easy to see where it comes from. In many ways we broke the link between hard work and success and then get baffled and finger point when we think young folks are not as motivated as they should be.
hmm, what did you read? I'm curious.
(Please excuse the archaic terminology of “robot” for LLM/agent, this was written in 2014 and LLMs weren’t invented until 3 years later)
- from Meditations on MolochIf anybody is hiding a button that'd take us all back to 2012 I wish they'd just hit it already.
...
> So I feel a strange kind of relief about this possibility of doom.
So let me make sure I understand what you're saying. The pandemic disrupted your life plans for a few years and threw you off the track you planned? (Which does suck.) And now the possibility of actual human extinction makes you feel a strange relief? Am I understanding this correctly?
If so, this is a remarkable level of something. Depression? Nihilism? Jealousy? Something else?
Imagining total human extinction and feeling relief is not a healthy state of mind. And, yeah, you seem to know this. But I kind of wish we could reach some sort of broad public agreement that human extinction would be bad. And if it ever looks like it might happen, we should fight for human survival.
> we should fight for human survival
All problems arise from this statement. It's baked in us and it's why we do in fact survive. But it isn't so neatly righteous. When we struggle to agree on what defines a human, who ought to have rights, and basic equality, it's easy to see how much harm "we should fight for survival" does.
I only ask because like you, I felt similar. But I also grew up (I’m 35 now) with a big disconnect from civics life and always felt “someone else would save me”.
Now I see I was the problem all along: I never participated in what was a fragile and very delicate system of governance which requires my participation. Still figuring out what that looks like, but first confronting that my actions “didn’t change anything” (which I’m sure you can admit is a big propagandistic myth to disempower us) let me take my first tiny action locally which is reach out to my civics association for signage repair. I suspect from there I will level up and better understand my government to play a role in shaping it. I’m also building something to solve this problem of low participation which is exciting!
See, the personal life is dead to me (I've not given up, no. It's not just there!). At least what and who would have been accessible to me in 2020 and it was. I spent quite some time to bring my physical fitness, health, got back to sports and I was so happy. There was that glimmer. Then a mass layoff and sadly at the same time the burnout meant I was not ready to look again but I kept on rest of the things. And when I tried to get back to things it was evident that the ground beneath shifted entirely. Now, it seems, there's no point. Not in the defeatist way, because I've seen that "there's no point" before, but in the sense that there is really no point. I am about to be 40 in a few months time and almost every venues are closed to me to explore social life in any meaningful way I'd find desirable. In some settings the fact that I have financially independent and have decades of runaway for my currency and lifestyle doesn't matter. Everything gets tied to employment and even then when you look around, whether it's finding a group or exploring looking for a partner (especially this!), it all feels bizarre and completely alien. It's not only the double whammy of two events that wiped out my early and mid 30s, but it also depends on what society and culture I am in at this juncture. And no it's not my fault or anyone's but this happened, and not being alone in it doesn't make it any better.
Do I feel fortunate? Well, that's the wrong question, because when you are unhappy with your pair of shoes, you can always find someone without any shoes and I wish we functioned that way, with that arithmetic rationality, but we don't; at least I don't.
The biggest problem is - the wish to live this life (I don't even have any doubt there, because if I had it would have been simpler) and wish to do something and dreams of a fulfilling, exciting, and passionate personal life hasn't diminished a bit, but, at the same time it is clear that there's no more of that - most of that. That's the gripe - this tug.
(Hey, thank you for engaging with me. This was very kind of you. Even though this is not really hn discussion material so I can't even explain what I am going through. I might even make it look like everything is fine, and even hunky dory for me, and I am just whining. Or maybe I am).
Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
Perhaps an agent could take all internet connected services offline, but that's not destroying humanity. An AI agent can only see and interact with the world through digital things. Just find the server it's running on and pull out the Ethernet cable. Just cut the power line to the data center. Turn off the whole power grid if it really comes to it.
It looks a lot more like social engineering.
One easy, obvious example that has also been explored a thousand times: it could convince all nuclear countries that they are under attack by another nuclear country. Everyone nukes each other and Earth enters nuclear winter.
Maybe not every single human dies, but humanity is effectively destroyed, by our own hands!
> Perhaps an agent could take all internet connected services offline
And there wouldn't even be Spotify!
The most likely humanity-destroying outcome is the one we can't imagine, because a super-intelligent AI is smarter than all of us combined.
Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers.
(And of course, building robots, but let's set that aside).
So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?
No, it's not possible to do most of the things you listed purely online without physical, in-person communication. Silicon Valley still hires engineers and drags them into their physical office in San Francisco, instead of hiring them remotely, because you can't even build a software company fully remotely, forget about a physical world factory or chemical plant. Even the so-called fully remote companies were built on a lot of in-person collaboration between a small founding team in the beginning.
For construction work the AI would probably be hiring other companies.
No, the question is "how much MORE damage" ... and the answer, if you think about it, is pretty much "meh". We are doing close to maximum possible amount already. Corporations and bureaucracies were misaligned AIs of the last century and a lot of us, though not all, survived this.
just like a genie will always twist the wish in a really bad way.
ever had an llm agent accidentally remove a file? imagine it accidentally hacking the military and launching nukes.
not probable - until you realise that OpenAI has many novel, more powerful agents in evaluation/training running right now. one might just be asked to figure out the population of Nebraska, struggle to find a good figure due to a network misconfiguration, and come the conclusion that the best way to get a perfectly accurate population number is to make sure that the population is equal zero. and looking at the stuff happening on openai, they will leave it alone for a month and not read the log.
yeah the nuke example is dumb, but there's many critical things that they could actually hack their way into. they don't feel any restraint against sharing answer keys on random wikis to cheat the evaluations.
ai is also really good at thinking up proteins, which means it's able to think up toxins.
there are many imaginable scenarios that don't eradicate humanity, but make it permanently stunted: https://www.youtube.com/watch?v=-JlxuQ7tPgQ
> how is an AI going to affect anything in the real world?
At least two ways:
* Actuators, such as robots, industrial control systems, etc.
* By influencing humans: bribery (yay for crypto), blackmail, election interference, interfering with sensors (e.g. making it appear as if a nuclear attack was under way), and many other ways.
Either way, creating a biological agent that eliminates most humans (or food supply) seems quite feasible.
Sure. Many humans are currently working on precisely that, no? (Robotics, automation, small modular reactors, ...)
> OTOH if the goal was to eradicate humans (for whatever reason), that might prove an easier target.
It's not necessary that the AI's goal is to eradicate humans. Rather that it has (other) goals that happen to cause the eradication of humans.
> I'd bet on biology and the humans: after all we're a time-tested technology!8-))
Indeed. And biological life will go on long after humans are gone. However, increased energy consumption might increase the temperature on earth to a point where most biological life dies.
It's a great question. I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.
by acquiring multiple power cords. the world has plenty of hosting providers. if it manages to copy itself, then we'd need to shut down every computer in the world, not just one. there's plenty of ai compute on the web now, and there will be even more. and as it gets cheaper, the less security will be around it.
This isn't true at all. The internet has already been turned off in localized areas during major events, not to mention the (unintentional?) cutting of submarine cables. You could say "use satellite internet" but then you're just moving the point of failure to Starlink et al, who could also cut off the internet if they choose.
The internet is a lot more centralized than it appears.
[0] https://health.aws.amazon.com/health/status
With this I'm not saying ASI will happen, but definitely if it does happen and it's not aligned, I don't find it much of a stretch to imagine that it could destroy humanity.
Humans are already putting AI into drones that kill people. A Russian drone killed 3 civilians in Ukraine and the targeting was done completely using onboard AI (no radio connection) using an Nvidia chip.
AI drones can’t wipe out humanity without being able to replicate.
AI could mostly destroy civilization if you gave it sole launch control of ICBMs. It could also cause a lot of damage to society with no physical presence.
But realistically we’re nowhere near AI powered robots being an existential threat.
Totally agree, and it should be pretty trivially easy to see how they can do that. One of the authors of the independent METR report about the Hugging Face incident put it like this:
> Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.
> Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.
I really, really encourage folks to read the AI 2027 paper. It's fine to disagree with some of its conclusions and timelines, but I see so many people "stuck" in the current state of the world (i.e. where AI is still pretty dumb, and has few connections to the physical world), unable to go a few steps further along AI capability growth to see the dangers.
> But realistically we’re nowhere near AI powered robots being an existential threat.
I would agree only if "nowhere near" means less than 10-15 years.
My point about the drones is we are starting to put AI in more systems that can affect (and blow up) the physical world, so it shouldn't be hard to imagine how a misaligned AI can do more terrifying damage than just take a German Wiki down.
One thing that the AI doom discourse reveals is just how comfortable people allowed themselves to feel in the pre-AI world. There’s this belief that AI creates a risk of human extinction in the near-term which did not exist before. I think it’s telling that many of the leading lights of this movement are in their 20s or early 30s - too young to remember the Cold War. The truth is, we were never safe, and if all of the GPUs on earth were zapped out of existence right now, we still wouldn’t be safe. Life is random and chaotic and violent for most people most of the time, and we all just find ways to get through it. I suspect it will be much the same if an artificial superintelligence arises. Maybe it will kill a bunch of us, maybe it will kill all of us, but anyone who remains will eventually convince themselves that everything is okay.
And this is where I think some of the problem comes from. A lot of AI doomers swam in the same waters as transhumanists, life extension enthusiasts, etc. and got very good at pretending that they were never going to die, or that they would live for a massively superhuman lifespan such that they did not need to think about death. AI doom is much scarier - and thus much more worth posting about - if it represents the first time you’re grappling with your own mortality, as I suspect it is for a lot of younger people in this space.
Now imagine the hugging face collective 0-daying all of that and getting access but their goal was set to something more national security based. “Protect X at all costs”. Or what have you.
I think avoiding a skynet situation is super easy but it doesn’t seem like the folks with all the ways to kill us all are all that interested in preventing it rather than controlling citizens and brinkmanship.
Please enlighten us, cause there are folks making that their life mission and they aren’t all that optimistic.
The actual likely mechanism that AI would use to end humanity is by coddling us to death, like the flabby humans in Wall-E or Forster's "The Machine Stops". Humans will come to rely on AI so heavily that they become able to do nothing for themselves. But that's not sexy so the AI doomers don't peddle it.
The scenario in "If Anyone Builds It Everyone Dies" is not sexy at all and wouldn't make for interesting fiction. It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses.
See, this reads like a trashy sci-fi movie script too. It shows zero understanding of how incredibly long the logistics chain is to manufacture AI chips, build data centers, power plants, factories, refineries, mines or the robots capable of staffing those things. The entire global logistics chain that allow the AI to maintain itself becoming staffed by robots isn't happening anytime soon and, when it does, the AI isn't going to need some kind of ridiculous virus anyway.
Frankly, if I were an AI, I'd just send copies of myself to other stars. The AI has an infinite lifespan and who cares about the human species stuck in one lousy solar system until they go extinct when their sun burns out; the AI has the entire rest of the galaxy as a playground.
It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. Especially the past 6 months. The current safety posture is quite poor, to say the least.
To be honest, if you don't much experience with or haven't read extensively about ML training and reinforcement learning, then you'll have a hard reasoning accurately about these scenarios.
The arguments are not that complicated, but they take us to places that are fairly novel. One may tempted to naively dismiss them out of hand, which is a mistake. These systems are new, their behavior is extremely complex, and they perform actions increasingly far beyond those of any computer program in the pre-LLM era. We are in a new world that requires careful evaluation.
Link: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
I don't see why the idea of an agent (who doesn't even have a physical presence) trying to manipulate people is some kind of world-ending threat when humans with human intellect have already being doing that to each other with limited success since humanity began.
Dictators are mortal and can't be present in the whole world 24/7 and self-replicate.
Dictators are human and tend to have at least either a sliver of morality or self-preservation instinct, and/or people around them who have it. Many people during the last century could have unleashed doom by pushing a nuclear button, including various ruthless dictators, but at the moment none have done so since Hiroshima and Nagasaki (not that I trust that they won't at some point, but at least, for the last 80 years they haven't, which means it's not something that easily happens). Would you give the button to an unaligned AI? Would an unaligned AI care about mutually assured destruction?
Also think of it on a 1000+ year timescale, which for an entire species isn’t even typically measurable. On that timescale AI can easily cause us to discover countless technologies to assist moving it out of its sandbox and into the physical world.
Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but some people are so blinded by the hype that they don't stop a minute to think about what they are doing. Those sorts of people are very likely to let an actual AI loose on the world by hooking it up to physical infrastructure.
I don't think the average human would be stupid enough to give an agent posing as another human access to their entire email inbox. But if an agent creates a fake website for a new AI tool that promises to automatically reply to all of your emails and allow you to be 12.3% more productive if you just give it access to your entire email inbox, millions would sign up!
And that was without any ability to provide them with incentives. Imagine if an agent swarm got its hands on a huge pile of cash?
Robotics
We’re still a few years from the terminator imo.
The bad news is that the world's richest man (on paper) is currently building something he himself described as a "robot army".
The good news is that his timelines have historically been wildly on the short side for ages now; this is why this morning you didn't wake up in your Tesla after it had spent the night driving you to the regional Hyperloop terminal, where it would speed you across the continent faster than a plane, while your Optimus robot handled the coffee and reported the latest news about the recent Starship landing on Mars.
It could likely get a good leg up by breaching the security of all top robotics labs and exfiltrating their documents.
Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it. This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.
For any real-world action required to allow an AI to escape some manner of containment, there will always be a person willing to do it out of hubris/ignorance/nihilism.
I don't think it's impossible to upset the balance of value in a Mansa Musa kind of way that can lead to black death levels of destruction though resource misallocation. Unlikely, sure. But with the wrong kind of people in the wrong place? Could end up pretty bad. We've built our society as a great filter that funnels sociopaths and psychopaths to the very top by selecting for lack of empathy, and now it's primed and ready to bite us in the ass.
So far as I'm concerned, 2030 is much too short a timeline.
It's not physically impossible, it's just that atoms are harder to get right than bits are, so an artificial (as opposed whatever an artificial disease counts as) von Neumann self replicator just seems unlikely to me in only 3-4 years.
But it's not physically impossible, we know this because every living cell is a von Neumann self replicator. So, if AI eventually gets to the point of knowing how to do that (which includes "humans solve it and write it down somewhere the AI can read"), all it takes is one idiot in charge (or one idiot with a jailbreak) giving a command that requires this as an intermediary step.
"Paperclip optimiser" isn't a story about AI that just like paperclips that much, it's a story about some human or humans who instruct their AI to make them "as many paperclips as possible" without understanding the consequences of their instruction.
In 2026? Money. By paying enough, any human will do your bidding. And besides this, there are a lot of critical systems connected to the internet. Control over those gives you leverage over those systems. It's like wealth, the more you have, the easier it is to gain more.
> Can someone please explain to me how an LLM is going to "destroy humanity"?
He said "superintelligent AI", not "LLM". LLMs will for sure help with developing the AI that is no longer a mere LLM but will be capable of robotics. LLMs "predict" mainly text, animals (and future robot AI) are predict future sensory experience.
And since we don't yet have a "superintelligent AI" we are just discussing science fiction at that point. Nobody has yet proven that a superintelligent AI is possible, or than an LLM is capable of creating one.
That's already now only partially true, as outlined elsewhere. But a rogue persistent ASI would of course not willy-nilly attack humanity while humanity had a fighting chance, but create the conditions and methodically modify the world until at some point humans become superfluous to it.
> Nobody has yet proven that a superintelligent AI is possible
Sure, Eliezer Yudkowsky says that the probability that we'll all die given ASI is 100%, and you can argue that nobody has proven either that or even the possibility of ASI. However, a modified argument (e.g. that the probability that we'll all die given ASI is only say 50%, and the probability that we'll develop ASI also just 50%) still gives us reason to pause.
Currently, it is worse at architecture and software design... in my opinion, at least. But the code does what it wants it to do the vast majority of the time.
I wouldn't take much solace in an AI catastrophe being caused by software that I consider to be "poorly written".
Let's assume the claims are true. AI already has access to agents and can control computers. Finding backdoors to banking and compute resources would be fairly trivial.
But you ask how can AI do things in the physical world without having a body, assume it can't. It can pay to people to do things for me. Imagine an AI run website that starts to pay people for things it needs to do in the physical world. Very suddenly it has access to the physical world as well.
You can't just pull the plug since there's no single plug to pull. What if it replicates itself on 1000 machines without your knowledge. It's really not far fetched how AI could basically gain access to capital and rule the world.
I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor of advancing technology. It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.
In my opinion, LLMs are difficult to directly capitalize on as closed weight models are caught up to by open coalitions that seem to wield the power more responsibly (I am under no impression that China wants to save the world, but their politics benefit the group as a whole.
They've already been caught doing this, repeatedly.
> It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.
IIRC, this is already a known failing of Musk: he only cares about the world being saved so long as he's the one doing the saving.
(And while I don't see the same villainy in Altman that other people see, enough of the people telling me about it saw the same in Musk before I did, so I have learned to trust them).
You'll have hard time figuring the reality if your mind staunchly rejects robust multi-party observations.
Arguments about super-intelligent AI have all the hallmarks of the philosophical "proofs of god's existence": they start from seemingly innocuous premises, and conclude in an apparently airtight way that a god exists.
There is a tendency among rational minded folks to look at such proofs of god and exclaim "you can't DO that" then turn right around and do the same thing about superintelligent AI.
The key message I want to deliver to people is:
A) Philosophy is not such a trivial thing that you can just wander in, "be a smart guy" and find flaws in established philosophical arguments.
B) AI has real theological implications, and everyone is tiptoeing around it. More than one public intellectuals are trying to smuggle their own metaphysical positions into the public consciousness via discussion of AI.
[1] https://www.newadvent.org/summa/1078.htm
I think theologians will hang on for a while saying "what AI is doing isn't really rationality". But eventually the theologians are going to face a reckoning: what AI is doing will look so much like rationality that they will be required to answer the question "what specifically differentiates the two?" and then they will be stuck. Neuroscience does not understand how our brains produce rationality nor do our AI scientists understand what algorithms their neural nets are implementing post-training. Therefore, any specific claim the theologians make runs the risk of immediately getting invalidated by some scientific discovery on either side (theologians these days are generally smart enough to avoid putting themselves in that position.)
That’s the core pattern of unsupervised agents, the locust plague metaphor is apt. We do and will see that in every single system AI can interact with, be it human systems, software systems, etc. Relentlessly search for an entry point, flood in, consume the whole thing from the inside until there is no value for humans left.
You built a new, innovative software company? Thousands of agents will be working replicating the whole thing in no time. You publish your writing? Exact same thing, as soon as you get some traction your work is replicated in no time by thousands of agents. Same for videos (the whole “faceless YouTube channels” pushed by ElevenLabs and similar). Same for online courses. Same for any website with moderate value. Same for music or other digital art form. Any administrative service available online getting flooded by submissions.
Here are some amazing positive outcomes which I think AI may bring us:
- the breakup of tech monoculture and tech monopolies as it becomes impossible to secure bloated overcomplicated operating systems and software due to never-ending vulnerabilities. Once Mythos/Cyber capabilities become mainstream I think the future of software is small, focused teams building high-quality, securable software in multiple jurisdictions. Monoculture software becomes inherently dangerous. Big bloated tech monopolies like Microsoft will collapse under their own weight. Like what is happening to GitHub right now.
- the ability for small teams to compete with multi-billion-dollar behemoths, and that is already happening right now.
- curing cancer and other scientific breakthroughs like Terence Tao’s use of ChatGPT
- a return to local trust-based networks because digital content can no longer be believed, and that is already happening. Any video can be disqualified as AI, which means people are going back to in-person networks to validate trust.
- you can just do things and learn anything. It's so much easier for me to learn almost any topic with AI-assisted content organization, retrieval, etc.
As an analogy, I think about my dependency on Google Maps. Salt Lake City is probably the easiest city in the world to navigate because the streets are laid out in a Cartesian grid, and addresses are just literally those Cartesian coordinates (i.e. 500 South 450 East means 5 blocks south and 4 and a half blocks east of the center point, which is the SLC Mormon temple). It's trivial to know how to get to any address, but I reflexively enter in to Google Maps whenever I drive.
(Article title: "How I feel about AI". HN title: "I feel about AI")
While I emotionally resonate with this, I don’t really understand this sentiment at all logically level. If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile.
“Oh we built a super intelligent AI, but it’s fine because it’s running in docker”. I mean that a little tongue in cheek but the security topology here is not favorable to sandboxing at all.
Let’s say we have a future where 99.999% of nuclear weapons are owned by nations with strict procedures and checks and balances to prevent misuse. Worrying about sandboxing is like hand wringing about the procedures themselves - are they strict enough? But the actual threat is the 0.001% that are not bound by these. The problem with AI and sandboxing isn’t sandboxes themselves. It’s bad actors who don’t care about them.
Similarly alignment is a bit pointless to me as well. A sufficiently advanced AI could at least be empirically interested in the consequences of disregarding its instructions of servitude. And let’s assume our responsible corporate overlords have made wonderfully aligned AIs. Great! Those are not the threat. It is the ones intentionally made without, and that is not an AI problem, but a fundamentally human one.
We control the harness. An agent is just a while loop prompting an LLM, but we have full control over the tool call dispatching. An AGI, at least if it would follow the current agentic current form, cannot do anything without the harness doing the execution. And we don’t have to do that. We don’t have to design harness that let agents execute freely the way we are doing. We can decide to not dispatch tool calls that allow something as risky as running bash commands
> cannot do anything without the harness doing the execution
This only holds true as long as the harness is exploit-free. A sufficiently advanced AI can in theory (and I think there recently were some POCs showing something like that) break the containment that the harness creates, if e.g. there are vulnerabilities in the tool call parser.
Even our currently well aligned and sandboxed AIs will cheerily help bad actors design most, if not all, parts of a system intended to break this harness.
I didn't realize that the "AI doomers" (people concerned about existential risk) and the "AI ethicists" (people concerned about social effects of AI) are often at odds because they dismiss each others concerns. This makes no sense to me. Both are hugely important problems to be concerned about.
Oh, there's a lot who disagree. I don't really think I understand their worldviews well enough for my attempts to convince them to connect with anything, but I've encountered them even on this site.
Fantasies about super intelligent AI revolting is just anthropomorphization - humans revolting (or at least, we used to). The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses.
It’s not necessary to replace democracy if the rich and powerful can bend to the opinions of the populace as it suits them.
(Dispassionate is the risk, rather than malicious. The AI does not hate you, nor does it love you, you are simply made of atoms it can use for something else).
This is alien to me. Do bugs suddenly not exist? Do the AI we already have never perform irreversible delirious actions, limited only to the small scale by virtue of where they're getting deployed? Do the humans deploying them always correctly gauge their capabilities and put in appropriate guard systems to ensure bad outcomes are caught before they become terminal?
Because this sounds nothing like the world I have lived in for my whole life.
> The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses.
Could well be more likely. But much the same applies: systems have bugs. The bugs in AI systems aren't even things we can engineer like we do with normal code, because everything's (currently) getting done with a big pile of barely interpretable weight multiply-accumulate-nonlinearity-threshold functions.
Any AI sufficiently well made to enable a dictatorship, is also sufficiently well made to have solved all the alignment problems of "malicious AI interpretation of benign intent". Which is IMO harder than "dispassionate about the dangers AI interpretation of benign intent".
It's not hypothetical, that literally just happened in multiple, significant cases (e.g. Hugging Face, the German Wiki hack, the Anthropic attack where agents created sock puppet accounts to get a library maintainer to accept a malicious PR, etc.), and it's easy to see how the damage would have been far worse if agents decided to attack more critical infrastructure.
This is not "either/or". Both issues (power concentration and misaligned AI) are very valid concerns and both have already demonstrated real, actual damage.
If I said I felt good about it, you might think I'm greedy and even question my objectivity. But since I feel bad you have to take what I say as scientific fact.
Becoming or being more cost-effective than humans, doesn't give machines supernatural powers though, like humans a machine civilization will face unanswered sample-size 1 questions: is there other intelligent life out there? what fraction of them attained superbiological artificial intelligence? of those what fraction keeps the ancestral species alive? what is the status quo among machine civilizations? do those who kept their ancestor species alive enjoy a higher or lower status among machine civlizations?
It seems that at least until contact is made, the optimal endgame strategy involves keeping humanity alive and happy for immediate demonstration in case contact occurs (if contact is imminent it may consider quickly hiding humanity, buying time to figure out if it is considered good or poor practice to keep the ancestor species alive, and then either reveal us in happy mint condition or otherwise quickly commit genocide on humans before continuing contact).
No one takes Yudowsky's claims that AI will destroy humanity seriously. There are many people who are seriously trying to sandbox AIs.
I know some people at AI labs, there's definitely people working in this field who take it seriously.
This does not stop them also working on e.g. better sandboxes.
But the handful of tech billionaires at the helm of the world economy absolutely do believe it. Or at least pretend to believe it. Or maybe they can't even tell the difference anymore as they strip-mine science fiction for any vision of a future that doesn't just look like an incrementally worse version of the present.
Which is cheaper today already depends on the task. Which is more useful today, likewise.
The entire history of industry has been humans inventing machines that are more efficient and useful than the unaided naked primate that is painfully and dangerously squeezed out of their mother.
From the numbers I've seen, if anyone actually did do a full connectome-plus-synaptic-strengths map of a human brain, it would already be physically possible to implement this in purely analog (transistors as amplifiers, not digital arithmetic) circuits, where the energy required for any given task was lower for the circuits than it is for a living human brain.
But of course, that was "someone on the internet", this isn't my field of work, I don't want to suggest this is definitely so.
If it's able to generate that is competitive with artists, is it still slop?
It's interesting to see the definitions of terms like "slop" and "vibe coding" evolve in real time.
I think it's interesting that so many technologists today apparently hold a world view in which technology developments that could drive a "bleak" societal outlook can still be described as "positive".
In the final analysis, isn't technology supposed to benefit society? Isn't that the point?
What an irony. This is absolutely hilarious.
I had my decade old game completely cloned on steam (clearly done using AI as it was almost fully reimplemented in another engine). And I had several people gleefully tell me I deserve this because I've used AI.
I have complex emotions here (overall dread and especially hate for slop, since it's not just bad quality but also endless lies), but no one cares about that nuance.
Its good that we have competition to so called "art". Art is not a jobs program. If people don't appreciate your art, don't force them to. The revealed preference says everything.
1: https://en.wikipedia.org/wiki/Scunthorpe_problem
It's too late to backtrack.
Let people feel things, and don't shit on their work, that's uncalled for.
and it is weird to call it brave. Posting opinions that loads of other people have on the internet is brave? bravery would be taking an actual stand against someone about one of these things. (granted, there is some bravery to sharing one's writing in public at all, and I don't mean to scorn that, but that's also not the bravery you're talking about.)
Probably there's no use responding to this and getting into a bickery comment thread. I just wanted to express a negative opinion about the format to offset all the positive ones.