Hugging Face: Anatomy of a frontier-lab agent intrusion

Don't miss their blogpost about the incident, which is long, detailed, and absolutely fascinating (but didn't make the HN homepage): https://huggingface.co/blog/agent-intrusion-technical-timeli...

Thread from yesterday: https://news.ycombinator.com/item?id=49089500

It’s an interesting post mortem and situation but it seems to say more about the weakness of the Hugging Face architecture than it does about the strength of the models.

This mostly reads like script kiddie style hacking, not some state actor black-ops stuff.

Man, these AI-generated UIs are so bad. They always look pretty at first glance... punchy headline, nice colors, lots of "widgets" and doo-hickeys. But it's just a nice paint job. Trying to actually read them is a recipe for suffering.

This one has at least 20 distinct text styles. They are seemingly deployed in random ways, following no discernible hierarchy. the smallest text is "9.6px" which is not only small, but also fuzzy due to the 0.6 pixels (?? why) making it impossible to read.

Likewise for the size, placement, and emphasis of various widgets on the page. Altogether it's just a big pile of information. It's hard to know where to begin, or how one's eye should move around the page

(In their defense: this particular UI is attempting to convey a very complicated sequence of interrelated events from multiple data sources. Doing that well isn't easy.)

This seems to be a colourful dynamic companion to this [1] blog post, which details the incident from hf's side. I recommend the blog post for clarity / ease of reading. But this one looks "movie hacker stuff" :)

Some interesting tidbits from the blog:

> While the intrusion did reach Hugging Face's internal infrastructure, the only customer content accessed was the set of ExploitGym/CyberGym challenge solutions stored in five datasets.

> self-referential search A quiet day. The main activity was queries to code-search engines and to the platform API for the identifiers visible in its own environment: challenge infrastructure names and environment markers.

> Self-referential search: code-search engines and our own APIs were queried for the names of their own infrastructure and environment markers.

I'm glad we got these details from hf, and not oAI. There would have been reasonable skepticism if they said that.

It's interesting because I think there's a chance the agent saw / inferred some hf-specific repos like blah/blahblah and "thought" that is part of the exercise as well. Everything they detail points to a probing and exploration with the sole goal to get those repos, as if it were on a hackthebox scenario. The few days between stages also might suggest that the "agent" noted this on one day, started doing other tasks, and returned when some of the tasks proved hard / didn't pan out, so went for the repos again.

The ease with which it got several lateral movements is crazy but not necessarily unexpected. If everything looks like a "test", you go for the lowest hanging fruit / most common patterns first.

What's weird about this is that there were several attempts to obfuscate / evade leaving plain text traces. I wonder if that is something oAI have specifically RLd for w/ scoring or if it "got" that from the literature.

It would be really really cool to see this from oAIs perspective (agent logs and such), but I doubt we'll ever get that.

[1] - https://huggingface.co/blog/agent-intrusion-technical-timeli...

three things jump at me:

1 - governments should be freaking out right now, because this tool could definitely wreak havoc on poorly designed systems.

2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?

3 - even though it's 100% open ai's fault - HF's design also seems silly to be honest.

If, like me, you're more interested in the actual exploit instead of the meta narrative, here's a relevant excerpt for the initial intrusion:

> Stage 1: reaching a launchpad by chaining through other parties' infrastructure. As documented in OpenAI's blog post (https://openai.com/index/hugging-face-model-evaluation-secur...), during an internal capability evaluation on OpenAI's platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.

Bottom line seems to be: So Huggingface has a feature where you can upload a dataset. The agent used that feature to insert an arbitrary payload into their environment. Now their environment also supports templates inside the dataset and the template ended up being evaluated into executable code

One thing I’m curious about: this was apparently a single multi-day run of an agent in an RL harness. What was OpenAI hoping to get out of this run? A single numeric score for RL training? A very long trace to distill into the next model?

A lot of people thought that OpenAI was making this up, and I hope if you believed that, that you recalibrate your opinions of what LLM's are capable of. Working with Fable and Opus 5 all the time, absolutely none of this surprised me capability wise, except for what seems like the long term planning capability (probably enabled by long context windows and launching subagents?)

I'm not shocked nor surprised by the incident. But I simply don't understand how Hugging Face is advertising this almost to the point of an "achievement". who does a step-by-step visualization to show how they were hacked? (outside of the likes of a Mandiant or Crowdstrike)

Does Hugging Face have a financial incentive in demonstrating OpenAI's model exploit capabilities?

this whole incident, while believable, still seems to me as possibly disingenuous.

This looks exactly like every other web UI built by Claude.

So let's get this clear: A frontier lab that fears Chinese extraction of their crown jewels, in Silcon Valley where every machine and VM is in full on employee surveillance mode because of competition and theft fears, A lab that's swimming in more funding than any startup in history, has been warning of the f'ing dangerous technology for 7 years now and with employees and CEO history of deception and lawbreaking left .... didn't detect a massive egress signature and the compute spikes from all these toolcalls coming from the research server hosting said crown jewels.

Go home Sam, nobody, absolutely nobody should believe this shit.