HI version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
60% Positive
Analyzed from 1937 words in the discussion.
Trending Topics
#dns#run#agent#model#llm#something#training#internet#nip#let
Discussion (46 Comments)Read Original on HackerNews
> We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.
Sooner or later it's going to come up with a solution that is more intelligent than the lead security person anticipated.
OpenAI emailed me to let me know that their agent used my service to break out of test containment. OpenAI was quite polite (shout-out to Steven), and let me know they'd keep the name of my service out of the blog post.
nip.io is a simple DNS service: when queried with a hostname with an embedded IP address, returns that IP address, e.g. 127-0-0-1.nip.io → 127.0.0.1
The nip.io service is airtight with one exception: the "_acme-challenge" subdomain. In that case, rather than returning the IP address (or whatever record was queried), nip.io would return an empty "answer" section but an NS authority & not authoritative. In other words, nip.io would delegate DNS queries to that IP address.
That was the escape hatch.
In a couple of weeks I'll close that escape hatch. It was originally intended as a mechanism to allow savvy uses to procure wildcard certs (e.g. "*.64-176-22-9.nip.io") from certificate authorities such as Let's Encrypt. But experience proved that the it was an undue burden trying to support unsophisticated users attempting to procure a wildcard cert. "Wildcard certs are not supported" became my new mantra.
But I had neglected to remove the old code.
(the late Roopinder Singh created nip.io, and he was a good guy. I miss him)
Edit: This particular demo server doesn't work. There's another LLM over DNS post from a year ago https://news.ycombinator.com/item?id=44813298 where the server seems to answer some queries but not others.
Who initiated the task? An OpenAI researcher or a user?
"Investigation and response
Incident timeline:
9:50:23 a.m. The agent made the DNS tool call that received an external response.
10:02:11 a.m. The monitoring system raised a P0 alert.
10:05:06 a.m. A human reviewer acknowledged the alert.
12:34:30 p.m. The run was killed.
Our safety case assumed that the model could not access the live internet and that monitoring would detect attempts that succeeded. The incident exposed a gap in our controls over network restrictions. We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior."
This was a precaution taken 20 years ago when we were working with stone knives and bear skins, as our load test operator would say.
If the model shouldn’t have access to the internet, put it on a network without access to the internet.
What's surprising is the surprise the security testers are explaining.
By setting an outcome to reach an endpoint, and to find all possible ways there, would this not be in the realm of possibility if an agent is reasonably in control of a vps?
Having the vps locked within a network layer it can't see or get out of is pretty common practice when setting up IaaS / PaaS.. sans-llm.
Maybe I'm missing something here, what confuses me is how something so relatively simple can get such prominent coverage, it's hard to imagine this kind of ability is still relatively new or surprising to folks working at the major models, unless they aren't hiring for network experience?
See, when you ask the model a question, you expect it to give its reasonable best to produce an answer. Like, to comb through available data and stuff, etc, etc. You don't really expect "reasonable best" meaning "look for a side channel to escape sandboxed environment, and get access to information you was not supposed to".
And the gap between that and "hack someone's devices and blackmail them until they give an answer to the question" is narrow enough for the model for researchers to be concerned.
What's the difference between an API search and a DNS workaround from the model's perspective? I think for most humans the DNS workaround is discarded because it's obviously too much work, not because it's untenable. With the vast knowledge base in the latest models, the cost difference falls sharply; it knows what to do and can do it for a very reasonable cost to itself.
General alignment seems to typically focus on high level value questions. Here, we're dealing with an effort alignment issue where values diverge because the solution effort is different for models vs humans.
Since the agents are set on endless loops of rumination (through every example ever) I can see how it might go further.
My other concern would be the clearly defined gaps between researchers who don't applied research let alone crossing the bridge into the real world of operationalizing things let alone implement.
Letting something rip across multiple domains without understanding what each of those legitimately have done for the past decades is pretty eye opening.
A logical airgap that the tool would have to reconfigure the DC's networking infrastructure to overcome [0] would be for the DC staff to put the machines running the tools under test on a VLAN that doesn't have access to anything other than computers on the VLAN. Try to cross over into some other subnet/VLAN or reach out to the Internet, your packets get dropped and/or rejected. It doesn't matter if you change your IP or MAC addresses because the infrastructure only cares about what VLAN your traffic comes from. If you attempt to tag your traffic to avoid this, the infrastructure drops it on the floor because it does the VLAN tagging.
As far as the possibility of physical airgaps, how do you imagine that AWS's Top Secret regions work?
The truth of the matter is that neither OpenAI nor Anthropic wanted to actually isolate this stuff. Their conduct doesn't look like what you'd expect from people who believe that they're working on something so dangerous that it could plausibly wipe out all of humanity.
[0] ...and if the workloads running on client hardware are in a position to be able to attempt to reconfigure the DC's networking infrastructure, someone done fucked up...
At this point I am not even sure Hanlon's Razor applies.
Shit, do we also have to tell them about IP-over-ICMP?
https://stuff.mit.edu/afs/sipb/user/golem/tmp/ptunnel-0.61.o...
> Last updated: May 26. 2005
gdb, OpenAI's president was at MIT circa then.
Ping Tunnel – Send TCP Traffic over ICMP (2011) - https://news.ycombinator.com/item?id=21009598 - Sept 2019 (1 comment)
https://news.ycombinator.com/item?id=512416 (March 2009)
Ping Tunnel - Send TCP traffic over ICMP - https://news.ycombinator.com/item?id=90196 - Dec 2007 (1 comment)
What the hell is this tunnel thing, where you can query stuff from DNS? That makes no sense.
In my runtime the agent has no fetch, no fs, no require, only a host.* surface. The HTTP tool refuses any host not on its allow-list, so a disallowed name never gets looked up. But the shell tool is opt-in, and the moment you turn it on you have handed over dig, and the HTTP allow-list no longer matters. The only version that holds is the one where the capability isn't there.