DE version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
58% Positive
Analyzed from 4120 words in the discussion.
Trending Topics
#virus#years#kill#don#more#doesn#need#things#author#something
Discussion (90 Comments)Read Original on HackerNews
I strongly agree with this. Too many people have the Arthur C Clarke quote infecting their brains.
So much of science is contingent on the physical world and cannot be determined from first principles i.e. by thinking hard about it. (Sorry Plato, you can’t think your way to the moon.) You need increasingly precise measurements, which means increasingly complex tools which means…supply chains. Science requires supply chains. Magic doesn’t.
These are some of the brightest minds on the planet, spending oodles of money on things they think they understand. And they didn't work.
I blame industrial illiteracy and post-literacy, but that's another discussion for another time.
Anything that can kill other people can also kill you.
I've posted this list before, and I will post this again and again until it sinks in, but this is a real list of real incidents that happened and will continue to happen as long as we study these entities.
If you are designing a supervirus that spreads via water, air etc., you better have one hell of a cleanroom, cabinets, manipulation equipment, incinerators (plural), hydrolysis and steam systems, multiple disinfection protocols...The list is nearly endless, and you can't handwavium it awway.
It doesn't matter if it's "all robots" or not. Even if it's "just robots."
If you're developing something that contagious, then it will start killing animals and it will spread. This has happened multiple times when organizations have screwed up maintenance, see this example from the UK,
It doesn't matter that the robots are out in the middle of nowhere.If you get sloppy, it will end up in the wastewater run off which will then get picked up and sequenced,
-A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.
It's worth repeating again,
Anything lethal enough to kill other humans is lethal enough to kill you.
And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.
I am writing a piece on the topic, please contact me if this is your field of expertise.
I am quoting as many primary sources as possible as it's not just some random on the internet, and I've gone back as far as the 1900s to find the many, many, many ways in which these statements are absurd, and misunderstands the potential of their technology.
Also, your own real-life examples show how "Anything that can kill other people can also kill you" has failed to stop people from tinkering with dangerous viruses! I'm not sure if you intend for these examples to be reassuring, but I'm not reassured.
> A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.
I don't think this is quite the argument. It's more like, standard graduate level knowledge _plus_ an insufficiently-aligned AI (which, being non-biological, is immune to viruses) _plus_ the ability to operate a wet lab via tool calls (it's coming very soon if not here already!) adds up to existential risk to humanity.
> Anything lethal enough to kill other humans is lethal enough to kill you.
It's also completely false; you carry tons of diseases that are capable of killing isolated islanders and incapable of hurting anyone connected to the general run of humanity.
This type of dysregulation also exists in AIDs. For the edge case of the islanders, then the level of risk and lethality our germs pose for isolated islanders with modern medical care is somewhat disputed, but it would be deeply unethical to ever test. We just don't know. So I won't pretend to know here.
However, these are edge cases that don't matter, because bioweapons, i.e. things deliberately designed to kill, just aren't the same as a dozen isolated islanders, astronauts on a mission to mars, or someone who is severely immunocompromised.
Weapons are designed to kill healthy people. The doomsdays being proposed are super-bioweapons that kill billions of humans. I think it's safe to say that as a rule of thumb, by and large, anything that lethal is lethal enough to kill you.
Just because I know the Kabachnik–Fields reaction exists and have studied it, doesn't mean I can do the Kabachnik–Fields reaction.
Just because you're smart doesn't mean the world will bend your way. Ask the legions who score high on IQ tests and end up bitter and alone; unable to achieve even a fraction of their goals (or conventional success).
The mortality rate from those diseases was higher than today because the population had no immunity at all, leading to both higher transmission rates and higher mortality rates. This would be true of an engineered virus as well.
And it wasn't just the one virus, it was an entire continent's infectious disease arsenal landing in overlapping waves. Again, something you'd do if you were trying to maximize damage.
So I don't find the bulk of the article about it being impossible for a virus to have this effect all that compelling.
I think the other flaw the article has it that it assumes the bad actor is doing targeted design. That they or the AI has some specific virus in mind meeting a set of predefined criteria and they are fighting the biology to hit that predefined target.
But what if the AI-enabled bad actor is a true chaos agent and has no defined goal and they are using tooling that AI helped them construct to perform genetic reassortment pretty much at random to see what happens?
Most of what they create won't really do anything. But some of it will. And there's a low but non-zero probability of Something Really, Really Bad.
I do not have the words how tired I am of hearing tech people and economists talk about biology.
AI will really aid the universe in generating more problems. Maybe it's not a supervirus, but hell, maybe it is.
"we could be attacked by an infinity of problems, which means that worrying about all of them is statistically unpractical"
The real deal needs a lab messing around with viruses and whatever happened a few years ago ... and it did not destroy humanity.
We live in a time of the fact off the fall of Pax American, the end of the START treaty, the reality of every country now needing their own nuclear weapons, increasing imperialism, and the possibility of AMOC collapsing in the next 20 to 30 years, plus much more -> the real possibility of polycrisis.
Given all of the realities, "AI doomerism" feels like a luxury.
My p(doom) us about 2%. That's a lot in my book.
(There are several biological threats the author hasn't considered. I'll just leave it at that.)
It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "superviruses" would require a lot of iterative experimental development in humans.
(And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)
Yeah, nice try ;)
Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships.
Again, most people just have no idea how hard - fundamentally hard - molecular physics is to predict, and how necessary experiments are for any development of biological systems.
If there is a fully general superintelligence - in the current state of alignment - we will all die for some reason. It doesn't really matter what the reason is.
is this a tautology?
(Also you don't even need superintelligence to do a lot of damage. In fact you don't need any intelligence!)
Sounds like there already exist some pretty crazy biological weapons in existence. I dont know if it will be 5 years or 20 years, but at some point some fanatic will try to kill the rest of us.
The doomsday virus has a first-mover advantage over the vaccine. The time that people start to drop dead from the virus is the time you start to make and distribute a vaccine. Even if AI can design and synthesize a vaccine within hours, how fast can it be rolled out to the general populace? It took months to distribute the Covid vaccines, even when we already knew how to make them. You have to physically load the doses on trucks[...]
was just 'make a viral vaccine with a shorter incubation period and infect everyone'. Handwavey problems require handwavey solutions.
It says no such thing:
> they design highly effective mRNA vaccines and antivirals against all the viruses and start shipping them across the world. It’s too late.
I think that's one detail that this analysis glosses over is that you design the virus to mutate and develop lethality. So initially, it's just like, oh, another common cold, but then once it's infected a large number of people, then it mutates and starts being lethal.
Firstly, it's not even clear that naturally occuring viruses can't be extremely lethal. There are examples (eg black death).
Even the existence of rare examples should be concerning, if attempting to argue against the possibility.
But even if there were no natural examples, it could just be because the ecosystem is in a balance, which them unlikely. That doesn't preclude someone from designing one.
To use an analogy. Natural predator prey ecosystems tend to be in balance. You could look around the world, before modern humans, and conclude that superpredators aren't a real threat, because predators that eat their prey too rapidly typically then die out. Thus most ecosystems you observe are in some balanced state.
But this doesn't mean modern humans can't come along, using their superior intelligence, and act as a superpredator. Ask the wolf population of western Europe.
If something is stopping us from doing this more often, it's alignment rather than capability.
You can't generalized from the typical behaviour that you see in a balanced natural ecosystem, and use that to say it proves there are no much more dangerous possibilities.
If the author made first principles argument about the inherent difficulty of doing something bad, even with abundant intelligence, I'd be much more reassured; or about defense offense asymmetry favouring defence.
What's here seems very naive.
It'll probably just hack and disable our water works, power stations and supply chains and let us all starve.
I'm confused. Would a bit of gain-of-function exploration + measles virus genome = supervirus?
I'm no biologist, but it seems like the genome is readily available. https://www.ncbi.nlm.nih.gov/nuccore/NC_001498.1
And it seems like the spots that have to be mutated are already known about: https://newsnetwork.mayoclinic.org/discussion/researchers-cl...
And this experiment has already been done: https://pmc.ncbi.nlm.nih.gov/articles/PMC8080110/
Five minutes of research suggests the sites -- Loop Epitope etc -- only need a few pairs of mutations each. this feels doable... It's not creating a supervirus from whole cloth, but why bother. Measles without a vaccine would be a catastrophe.
EDIT: actually it isn’t even that. I personally don’t know enough about this topic to really have an opinion or criticize anyone. Still it’s indeed a phenomena I’ve noticed.
This is such a massively overblown worry that AI companies who are seeking regulatory capture love to roll out whenever they are trying to gin up some FUD.
Creating viruses at home is extremely difficult, even if you have a perfect instruction manual and someone sitting behind you telling you what to do. It's full of trial and error and validating each step requires rigor.
It's also a process which requires fairly expensive and specific supplies, which can be easily gatekept. Instead of trying to gate-keep knowledge of biology, we can do things like KYC around specific types of lab supplies (like we already do with fertilizer which can make bombs, or specific types of supplies needed to make nukes). Heck the Biden admin already added a KYC requirement for plasmids and plasmid companies already have implemented screening for sequences which could be dangerous.
It's even happened pre-AI https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack
Let's look at it in an inverse way. Making a single drug that impacts a single protein, intelligently, from the ground up, has arguably never actually been done. Despite unbelievable financial incentive. And with AI.
So if I have to choose between the slim possibility that a crazy person might develop a bio-weapon and the possibility that Anthropic, et. Al convince the government to regulate and control open source models (because I'm sure the government will only do that for good reasons and never things like covering up scandals, squashing dissent or enhancing their own power) and any competitors. I'm probably going to put my chips on a nutcase in a trailer park developing super sars over regulator capture by the government.
Again the same logic applies though, it takes significant effort and money to create sarin, it requires a lot of rigor. That is the limiting factor, not the knowledge of how to make it.
If we are worried about people making sarin in their garage we can do many things to control the supply of chemicals needed to make it. We don't need to try to restrict knowledge of chemistry.
"regulation of current models" basically implies that open source models will be heavily restricted if not banned outright and guarantees that intelligence will be owned and gatekept by a duopoly.
A major focus of the article is about the fundamental tradeoffs in how viruses operate. Was that not enough for you?
To be clear, I am not saying that such a virus will exist eventually. I am just noting that the arguments displayed here are not completely tight, and there are unknown unknowns at play. Some things are unfalsifiable. This is common.
> PhD students in 2026 using state-of-the-art AI software are spending months or years trying to design simple peptide binders that inhibit some enzyme or pull down some protein
This confuses me, as state-of-the-art AI software is moving quite quickly. Any software that is currently state-of-the-art AI.. hasn't existed for years, maybe not even for months.
It also seems quite odd to claim that there are no sufficiently dangerous diseases when we have high-security research labs dedicated to exploring.. exactly these kinds of diseases.
And, even if it's not a disgruntled teenager, it just takes one leak from a lab that has 10xd productivity using AI tools.
This is a real risk that existed before AI, one that I think we can now confidently say that society is under-prepared for since 2020. It is certainly magnified by AI, even if it's not 100x.
The author even seems to recognize this:
> So, if you’re worried about AI-designed superviruses, think COVID, not Ebola. And if you’re now a little less worried, maybe you understand where I’m coming from.
This is exactly where I think most people are starting from, and that's _why we're worried_.
But it's pretty clear you have not?