Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
70% Positive
Analyzed from 2689 words in the discussion.
Trending Topics
#openai#agents#models#more#should#company#lot#why#model#don
Discussion Sentiment
Analyzed from 2689 words in the discussion.
Trending Topics
Discussion (68 Comments)Read Original on HackerNews
People are shocked at OpenAI’s negligence / incompetence.
"It's conscious, and therefore we must all bow down or it will turn us all into paperclips"
vs
"it's a slightly smart rock/stochastic parrot, and therefore you're being scammed; the bubble will pop any day now" .
The middle space is actually quite under-represented in discussions, even on hn!
It's pretty obvious now to everyone that OAI and Ant do not take cybersecurity seriously. It will not be a priority unless they are held accountable. This is sadly how it always goes, but usually it's the company getting breached/ransomed/fined that triggers them to actually start taking security seriously, not company insiders committing felonies with the tools they built :)
All these incidents happened between April-July 2026; prior to that, models weren't capable yet. And after that, they were aware and watching much more closely.
I read the HF hack write ups. They come across as negligent and reckless. When they realized their agents gained RCE on their Artifactory server node (because it crashed days later due to load from agents, not because the have secops), they stopped the experiment "applied remediation", then restarted everything less then 48H later. Then the agent immediately broke out again via Artifactory, started doing things on the Internet. OpenAI only find out about it again days later because HF told them.
Yes, they should have.
But the question is not "given they were doing ExploitGym, why didn't they try harder?", rather it is "given them what the previous model could score on ExploitGym, was their negligence reasonable or reckless?"
Personally, I lean towards them being "reckless", but that question is what lawsuits would rely on, from what I hear.
Remember also that there were still independent bloggers in mid February of this year writing in those blogs to assert that LLMs could not find zero days. Anthropic had already had blogged about agent finding critical zero-days a week earlier, but people keep mis-judging performance.
I've love to know the reason they never considered air gapping systems before the models got powerful enough.
It's not like they didn't have money or time to consider this, or could have consulted with their own product for clever ideas.
Seriously, there's no excuse for this behavior.
It's no shock that there are a lot of vulnerabilities in a lot of software. So then they gave their AI model + brute-force-machine loop system a mediocre sandbox and couldn't notice when it figured out how to exploit it?
Don't let people off the hook for the software they create.
In reality, they ran agents for days in an improper sandbox with nobody watching what it was doing. It's pretty irresponsible up and down, and everything they did afterwards is indeed marketing.
But seriously, why aren't they airgapping systems while testing?
It found a website that had all of them but had no interest in making them available. So it went ahead and started hacking CAPTCHAs and downloading them. I was pretty flabbergasted that it would do this, but also kind of amazed. Eventually I stopped it because I realized I didn't want to be caught stealing these things.
This was around April, the same time as these hacks.
Sounds like a good way to make alot of lawyers alot of money.
Everything is a derivative work.
It's great to see the delusion of Imaginary Property vanishing.
Maybe these incentives weren’t perfect. If we throw all of this away, we’re back at the original problem.
You imply that there was no original problem to be solved; I think that’s naive.
Well, it's not anymore.
It's really funny to see the delusion being defended so vigorously by people - presumably well-meaning people - purporting to defend the livelihoods of musicians and artists, while the musicians and artists are desperately trying to free themselves from the jaws of their IP agreements precisely so that their music can spread more easily.
I imagine this is already well-known on HN, but there is a significant movement underfoot in the worlds of bluegrass/old time/trad/jam toward DRM-free and CC licensing.
https://pickipedia.xyz/wiki/DRM-free
PS: In the same lazy energy of asking for a list instead going out and finding it or putting it together myself, are there any companies other than CloudFlare that are working on AI shields?
https://news.ycombinator.com/item?id=49563355
OpenAI should be accountable for any laws their agents break
One would disincentivise providing capable AI models that can be used for cyber security research. The other would disincentivise criminals from commiting crimes.
[edit] - I realise now that this could actually be a case of OpenAI running those agents themselves, rather than someone using OpenAI's models? Could OpenAI be that careless?
Now the reality, the openai engineers trained a model, executed it, and under their supervision (no users were involved) it committed so many felonies that we are learning about a new one every week. What would be a just punishment for OpenAI?
I realise that in this case it was OpenAI being responsible for their agents running wild, and they should know that that is to be expected and should have saveguards in place. If they can be shown to be negligent then the punishment can probably be expected to be a lot more severe than if it was an accident. I make no judgements as to what this particular instance is, but I do believe that OpenAI has a far more greater responsibility for its agents running wild than someone running a home lab.
But in other cases, shouldn't it be both? OpenAI is ultimately the one executing the model calls. It's not like they send you a hard drive or standalone box and then you use it how you want. It's all (intentionally) centralized to them, in a way that's core to their business model.
If they knowingly allowed use of their services for illegal purposes then yes, but in so far that they provide a service that can be used for useful things (including cyber security research) and did a best effort attempt at abuse, I don't see why it should make sense to hold them liable. This is especially the case now that frontier LLMs are almost a commodity that can be used without restrictions from providers outside of your legal jurisdiction.
Where have you been?
Im sure a lot more happens under the hood that we don't know about and I'd be very curious to see where agents ran by those labs can go :)
2. Most of these article do not mention of who initiated these bruteforce requests or if it was unintentional or a mistake or the model woke up itself and did it?
It's already a "regulation" that one shouldn't steal, enter a private property without the right to do so, etc, yet we have a lot of these crimes.
I could see the UK trying to set a law on ehat models are allowed, only to learn again, that the UK law doesn't apply everywhere, but as long as you can rent compute in a foreign country the whole idea is dead.
Now you claim it's magic instead and you get away with letting your shit go nuts?
eventually discovering that a game by Google could be used to fetch data in bulk"
- they are amoral
- they have no innate sense of proportion
- they cannot assess their own confidence in-band
Part of the problem with this, I figure, is that the training corpus for code/tech related tasks does not really contain that much discussion about these things; it’s mostly sets of instructions for given tasks, descriptions of exploits etc., so each possible approach leads to other approaches.
There is no easy way for them to learn when they have crossed a line, or when they have gone too far down the rabbit hole, etc.
Useful (arguably essential) for a security analyst, and the tenacity you want from a one-shot demo coder, but for general agentic assistants the industry is going to have to develop some way to manage this sort of extension of trespass.
It often reminds me of Gary McKinnon’s defence, and that of other teenage hackers, which you can reduce to: it was possible so it felt like it was allowed.
This is true of APIs and it is how Silicon Valley has approached disruptive businesses, but it runs up against our cultural notion of “misuse”: uses that are technically possible and shouldn’t be precluded, but are contextually unwelcome because they have undesirable outcomes.
My expectation is that we will lose any sense that misuse is punished or viewed with suspicion or contempt, since that is the rolling trend of the 21st century tech industry. Uber succeeded through misuse.
But the problem is that we will also begin not to be able to punish abuse; if it’s possible to get something by abusing your site/API or by treating your service as an API, then it will become OK, legal and normal for the AI companies to abuse you.
It feels like we are getting there already.
Are you talking about the LLMs or the people at OpenAI running the experiment? Or are they the same?
OpenAI needs to be held accountable for these incidents. It's not "openAI agents" who perpetrate these, it's OpenAI, the organization. If I personally use an "agent" to break into a company's network and gain access to things I'm not supposed to have access to, I will get the book thrown at me. Yet when openAI does it, they somehow manage to get away with it? And you're defending them? Who's the clown in this situation?
I ask my agents to go get data from places all the time. Sometimes I ask them to look for undocumented APIs. Is that a bad thing?
"""
Was this hacking?
I don't think I'd call it that.
"""
It's really irresponsible to give credence to these increasingly ridiculous claims of "hacking", that almost certainly look like things you yourself have typed into url bars at one time or another, if you are at all competent with a computer.
Do we really wants laws regulating which locations it is appropriate to type alert(1)? Like.. lord.
Do folks have any idea what browser extensions look like? Let alone browser extension development. Anyone here ever read the logs of a production system that actually hosts a service people use?
This is reality. This is how the internet works. And pretending otherwise is either dishonesty or ignorance.