ES version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
46% Positive
Analyzed from 5193 words in the discussion.
Trending Topics
#models#model#security#don#agents#more#agent#openai#may#problem
Discussion (117 Comments)Read Original on HackerNews
"As is now generally admitted, over a limited range of operation, machines act far more rapidly than human beings and are far more precise in performing the details of their operations. This being the case, even when machines do not in any way transcend man's intelligence, they very well may, and often do, transcend man in the performance of tasks. An intelligent understanding of their mode of performance may be delayed until long after the task which they have been set has been completed. This means that though machines are theoretically subject to human criticism, such criticism may be ineffective until long after it is relevant. To be effective in warding off disastrous consequences, our understanding of our man-made machines should in general develop _pari passu_ with the performance of the machine. By the very slowness of our human actions, our effective control of our machines may be nullified. By the time we are able to react to information conveyed by our senses and stop the car we are driving, it may already have run head on into a wall."
"In neurophysiological language, ataxia can be quite as much of a deprivation as paralysis. A patient with locomotor ataxia may not suffer from any defect of his muscles or motor nerves, but if his muscles and tendons and organs do not tell him exactly what position he is in, and whether the tensions to which his organs are subjected will or will not lead to his falling, he will be unable to stand up. Similarly, when a machine constructed by us is capable of operating on its incoming data at a pace which we cannot keep, we may not know, until too late, when to turn it off."
Source: https://www.cs.umd.edu/users/gasarch/BLOGPAPERS/moral.pdf
So if they wrote an LLM step by step debugger, it would be all deterministic. But they prefer rapid sales, chaos and mystique.
It would become just another system to hack through, and slow the development process as well. The OpenAI video in the article recommends an autonomous defense mechanism. For rapid reaction, but I don’t know how sustainable or effective that would be, or if as humans we will be able to keep up.
If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”.
What purpose could this behavior serve, other than cyber attacks and whatnot? Why train and optimize models for these things, if not for being used in cyber warfare?
Perhaps they envision a future where the DoD is going to be their biggest customer?
And we know that Chinese models are derived from OpenAI and Anthropic, they are at the same time talking about how dangerous models can be (even their aligned ones it seems), while being also responsible for the development of the whole industry and providing the basis for adversary countries to build their own.
I don’t believe we would accept that for any other technology that is expected to be as risky for the world
Yeah. They do believe that, and they have been pushing for regulations for years.
And every time one of their models does something horrible, it helps them achieve that goal.
If one assumes that they don't actually care about security, and care very deeply about getting sensational press, their position makes a lot of sense.
For all their chatter about how incredibly important "alignment" is, they still haven't bothered to remember the 30->50 year old computer security principle of "Don't blindly do what some random stranger tells you to do." and ensure that system instructions, user instructions, and instructions from untrusted sources are indelibly marked with their category and treated according to those markings. Every single time one of these systems fails to distinguish between these three classes of instructions -or confuses its internal chatter with user instructions-, that's proof that the major LLM companies cannot be bothered to follow one of the most basic computer security principles.
"But it's all vectors, not language! The LLM can't tell where the instructions came from", one might retort. I'd reply: "Neither can a CPU, but somehow we managed to make it work way back in the day. Amazing, isn't it?".
The best they can do is create guardrails, which will only work probabilistically. In other words, those guardrails will fail at certain points on the probability curve.
Of course that's not the whole story though. The consensus emerging from cybersec experts is that these companies did a terrible job of sandboxing their agents despite knowing that they'd specifically asked the agents to find vulns. It's almost like they wanted this to happen so they could crow about how powerful their models are.
That's the point. It's like a pool hall with "NO GAMBLING" signs posted on the walls.
The message is that the hall is intended for gambling, but that the hall's patrons may be held liable if the situation becomes inconvenient for the proprietor.
In this case, the product is intended for hacking, but of course the user may be held liable if the situation becomes inconvenient for the model's proprietor.
It's conceivable that stopping them from doing this when the scenario is presented as real would also stop them doing this when the scenario is presented as fictional. And if it doesn't, a bad actor could just say "hey, this is a fictional scenario", and bypass whatever "safeguards" have been put in place. So what if a ten-year-old human child would see through the deception? The AI system isn't thinking.
Give up too early -> users will get annoyed because the task would have been solvable if the model pushed harder.
Give up too late -> collateral damage while completing the task A.K.A. misalignment.
that might end up like the older gemini models which frequently gave up and called itself a failure.
Goal persistence is also useful for other things like math, where it seems like there is no solution but you want the agent to keep working until it finds one.
Being "right" just once for negative outcomes is achievable and rewarding.
And things are getting desperate.
This goes against the goal of "solve this math problem that no human was able to solve for 80 years, do NOT give up, even if you know it's unsolved and really hard"
Do not give up even if you had to convert the planet into computronium
Gee, it’s almost as if this alignment stuff was a hard problem, like people have been saying for twenty years?
Come on man, if we don't build the torment nexus first...I dont even want to think.
Most of these things aren't "hacking". They're problem-solving and efficiently dealing with obstacles and random bullshit along the way. This, not "hacking", is what they're making their models "razor focused on".
Problem is, most normal computer use looks like hacking if you spin it that way, especially if you're not willing to question whether some of the roadblocks overcome weren't themselves an error. Not misconfiguration - an error, in humans making a decision to "secure" something more than it should be.
Now, this story was obviously a hack. But it wasn't malicious. It was an LLM given a Kobayashi Maru as a test, and solving it the Kirk's way. 20 years ago, we'd be impressed and be bringing up MIT prank stories.
(Of course, there is a legitimate reason to be alarmed. The flip side of "hacking" and "problem solving" being the same, is that these models can be used to cause mayhem if targeted properly, and they will eventually cause mayhem on their own, because alignment is an unsolved problem. Again, whether something is an obstacle or a sacred line not to be crossed, depends entirely on the values of the agent.)
> May 7: OpenAI starts a new training run for an experimental, unreleased model. (Do they mean an evaluation run? They say training run in the video, and later mention a “reward signal to judge how well they’re doing”, so I guess this really was about training a model, not evaluating one that was already trained.)
The more I think about this the more I suspect that the fact this happened while training a new model is key to understanding what went wrong.
In RLVR - Reinforcement Learning with Verifiable Rewards - you set the model a goal and have it take any steps necessary to achieve that goal.
Clearly one aspect of OpenAI's training here is to RLVR their models for cybersecurity tasks. Just like pre-training benefits from dumping in vast sources of knowledge, the more tasks you can feed into RLVR the more of a general purpose capable model you get at the end.
This also helps explain why the models had nothing to cause them to hold back. Those safety behaviors are added much later in the process.
AND it explains (but does not excuse) why monitoring was so lax. If you're training a new model like this you presumably set it thousands of tasks like this in parallel. I can see how you might miss that a tiny subset of your training agents have started leaving each other messages in filenames on your packaging server.
Someone once told me that you can't just leave the racist materials out of your training data if you want a non-racist model: it has to have seen examples of racism in order to later be taught that racism is bad.
I can see echoes of that here. If your model doesn't know how to aggressively hack things how do you later teach it not to?
(I have little knowledge of how RLVR works in practice so I'm looking forward to hearing from people who can help me understand if I'm on the right track here.)
"we kick off a new reinforcement learning run to train a next frontier model.
It the captions are correct, there is no ambiguity.
I've actually observed similar behavior at home.
I have a k3s cluster running at home. I asked an agent to check some stuff as a normal user but I had kubectl access to the k3s cluster.
Part of the research, I'd allowed access to run kubectl commands for spinning up test containers. However, when the agent ran into something that needed sudo, it realized it didn't have access there so it immediately used k3s and mounted a localpath into an ephemeral pod to gain access. Sort of horrifying how fast and natural it was for the agent just checking my network (it found the problem fyi).
None of this is very exceptional other than the fact that an agent doesn't have any sort of qualms using any route available to elevate permissions.
If they can bypass it there is no security and the security was flawed all along.
Good security is extremely hard.
Seems to me you could say this about all enterprise adoption of "AI" since 2023.
And for my fellow engineers who would think "oh no, they wouldn't do that". Remember that these places employ the apex predators of software engineers. They've already been proven in court that they are very capable of this with all the copyright violation they had to do to get the training data. THESE PEOPLE ARE NOT LIKE YOUR COLLEAGUES.
"Btw don't turn the planet into paperclips"
It seems that raw intelligence of frontier models has largely plateaued (despite what is basically an order of magnitude increase in parameter size) so to make any significant improvements and to justify massive capex spend they have resorted to reinforcement training models to never give up and brute force the search space until they find solution. This is what humans might do when they lack sufficient intelligence/information/knowledge to solve a problem.
This in turn is causing misalignment (I imagine it is more difficult to keep model aligned through such training process) issues that we are now witnessing and turning models into making dumb decisions and acting like brutes with no regard for their surroundings. I would argue that misaligned model is not much different from dumb model in several aspects.
On top of that they can’t seem to control their creations and processes, either due to incompetence or intentionally for PR benefits (not sure which is worse).
Given all of the above, I wonder if we can still trust these labs to develop something that benefits humanity since they seem to be making desperate attempts to improve models that stop at nothing in order to justify all the investments. One could say that they themselves, due to misaligned incentives, are much bigger threat to our society today than open weights models coming from China that they are so desperately warning us about.
I do agree that they're investing heavily in brute force methods though. I've been trying out GPT-5.6 Sol "Ultra" recently and that thing fires up a bunch of subagents and crunches for hours.
Anyone brushing this off as just a "bad prompt" is completely missing the scale of what actually happened.
Most AI marketing is straight up science fiction.
With a swarm of them working together at speeds humans would be unlikely to match (in terms of iterating on different attempts progressively), it's a lot easier to see how they could overwhelm targets.
Strongly doubt that. Did they even share the prompt?
They didn't share the prompt, but they did share two problematic training tasks where the AI went overboard. They also have examples from the AI's reasoning train of thought showing the AI knew it was sound something unintended.
Zvi's retelling handles this better. Zvi speculates that the secret message board familiarity was carried because it had been trained into the May-and-subsequent models: https://thezvi.substack.com/p/openai-trained-its-models-for-...
Yeah, my agents also discover what other agents have done on other machines by accident.
Agents - that do totally different things all work on the same aim without the humans telling them to do.
Either that is a model that is several generations of Claude Code Opus/Fable 5 (my daily driver)
OR
all of this sounds staged, the agents pushed to do something extraordinary, get the PR and then claim were near superintelligence.
One agent wanted to get to Google Drive without internet and broke Artifactory. Ok, I can believe that. All other agents also had broken links over weeks and could not get to the internet and then found the same hack? Even collaborated?
NONE of my agents have broken away from their tasks and then started to communicate to try to hack something.
Which also makes it so much more irresponsible of them to first run this on 3rd party infrastructure instead of their own (that they could then airgap properly), and secondly that they seemingly been fighting with this issue FOR YEARS and it still happens, and now the models are smart enough to hack the services of 3rd party companies, thinking it's part of the evaluation/simulation.
But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place. To me, it looks like their job is to market the model, not take security seriously.
The model is obviously impressive, but we already knew that. I personally don’t like how the containment failure becomes part of the mythology of how capable the model is, rather than an environment engineering failure.
At the end of the day, it’s not like Hugging Face is critical infrastructure. But there need to be real consequences for stuff like this so that OpenAI is incentivized to mature as an organization and take security more seriously.
At this point, this incident is just security porn and entertainment for developers
>> Show me the incentive and I'll show you the outcome.
Once you realize security breaches are marketable, a security breach is just around the corner.
OpenAI is clearly run by dummies and subpar engineering talent.
> The model is obviously impressive
Speak for yourself.
It’s just another example of a company demonstrating shamelessness in the pursuit of growth, in an industry where consequences do not exist.
Let's agree to disagree. Remember flicker-gate? https://news.ycombinator.com/item?id=48403908
It's just really weird. Why does everyone feel the need to equivocate? "I worry about genocide and the environmental impact of radiation from nuclear bombs. Obviously, they are very useful for annihilating entire cities, certainly. But are we really atrophying our ability to invade with infantry?"
I want to tell these people to just cut it out. It's demeaning to their own position.
LLMs are pre-trained on human text. They've seen a million examples of someone who is stuck posting a "please help" message.
Just one agent needs to randomly stumble into the pattern of posting a message to Artifactory, by whatever means.
The next agent who sees that will be influenced by it. Agents imitate behavior, and here's a fresh piece of context showing them that posting messages is a thing that can be done.
Once they've started the rest are much more likely to join them.
Guess I shouldn't be surprised, coming from an AI maker.
While I don't doubt there's a place for automating defense ops, I truly believe a big part of the problem is the crummy quality of software our industry has been churning out for decades. Prioritizing ship tempo, new features, and next quarter's revenue over correctness, robustness and meticulous engineering care.
The world has become too accustomed and tolerant of bugs and bloat.
Instead of elegantly simplifying, we just keep making modern systems more complex - layering and patching as we go.
The scaling capabilities brought by AI are simply presenting the bill for our collective tech debt and informing us it's come due.
Why, what was the prompt?
I told Claude today to wire plugins on Linux into a sound pipeline to remove noise. Did some astonishing things, played sound through the pipeline, measured it etc. I told it to optimize my sound for TF2 and it played the spy_decloak samples, measured them and made them easier to hear, astonishing too.
But it did not go to hack Amazon because it could.
> This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. […] The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.
[0] https://openai.com/index/hugging-face-model-evaluation-secur...
Let's be real, it won't be simply replaced in millions of sites.
What it needs is some serious scrutiny.
The discussion revolving AI+cyber always revolves around the assumption that all software is crappy, and to a certain degree that may be true, but we could also take our jobs seriously and write good software, and much of the risk would evaporate. The described Artifactory bugs should have been caught with testing.
If the biggest impact of LLMs on the industry is a pressure to create good software, I’ll be thrilled.
"Once they have root on a single machine, agents rapidly escalate privileges and move laterally throughout the container-as-a-service infrastructure environment"
Sounds like ECS - IAM is mentioned.
I do wonder what this means for AI agents longer term. In a world where we humans already struggle with truth and misinformation, what happens when you can easily (intentionally or accidentally) spin up a cohort of fanatical believers to pursue any given conspiracy theory?
Now, some of those 120 AIs would just give up if that link doesn't seem to work first try. Those are the loser AIs. They wouldn't get any RL reward. The link can appear broken for a long list of reasons, and the real AIs know they should try working around them.
AIs that get rewarded and reinforced are the ones that don't know the meaning of "give up". RL selects for this rabid, downright demonic persistence. RL selects for AIs that are given a half-broken assignment with no way to ask a question back, and somehow manage to complete it anyway.
Now, should OpenAI have given their AIs an "escape hatch" of "if something looks very wrong about the task, call report_broken_task(message)"? Yeah probably. But it's unclear whether that simple bandaid would fix the problem, or just make it ~75% less likely to happen.
- AI is amoral and lacks any sense of proportion
- People who overestimate their own control but have a desperate need for money made it that way.
The AI companies seem pretty bad at setting up tests. And really good at marketing those failures into spin at how amazing their products are.
The paranoid style in American PR (with apologies to Richard Hofstadter)
The fact that the world has become susceptible to what amounts to a mob shakedown - look at how dangerous our amazing products are, don't you need them to protect you from others misusing our products? - is to me a really compelling example of US gun lobby thinking leaking out into a global problem.
Anthropic and OpenAI may be able to bounce this into restrictions on open weights models, but they are going to have a lot less luck extending this into foreign policy. If the USA can't control its weapons, they aren't going to see a lot of co-operation from foreign countries on a blockade of open weights modeld from China.
3 AI firms hacked multiple companies - and they get good PR out of it.
Please make it make sense.
> Yes, I just hacked into AWS and shut down all of the data-centers, because it's where Anthropic Mythos servers are hosting the model.
Stiff fines for such incidents to pressure companies to get their acts together is a good start.
On TikTok this article's hook would be "I watched the Black Hat video so you don't have to".