Pacing model development in an era of cyber-critical capabilities

Some more info in a Wired article [1] and quotes from Sam Altman to Alex Heath [2]. The official blog post says vaguely "The signals we are seeing from upcoming model progress make clear that we need a broader approach", but the quote from Sam Altman explicitly says unreleased models are showing "various degrees of misalignment".

This is also significant - pausing frontier training runs for multiple weeks to ensure agents are sufficiently aligned and avoid another rogue agent situation:

> This included a two-week pause in reinforcement learning (RL) training on our latest models intended for deployment while we further hardened and red-teamed our research environments and expanded the coverage of our monitoring systems. Our largest planned frontier RL run remains on hold while we conduct smaller-scale training and evaluations to assess model behavior, validate our safeguards, and establish more evidence of alignment before proceeding.

[1]: https://www.wired.com/story/openai-overhauls-safety-protocol...

[2]: https://sources.news/p/openais-big-slowdown

When science fiction writers imagined the development of superintelligence, it was on air-gapped networks with strict access controls around it. They failed to anticipate the competitive pressures of capitalism...

We need strong AI safety regulation yesterday. And unfortunately it's not enough for it to be just national regulation; we need international cooperation on the matter.

There’s one thing here that I’m really curious about, and that is what happens in between detection and the decision to pause. Basically, it’s about monitoring any system and the authority over its actions. For humans, 30 minutes to investigate might be considered reasonable, but what if during an investigation there’s a high-risk tool call? If the tool execution happens in real time, then the monitoring becomes retrospective, and if the execution is held, then monitoring latency and uptime are a part of the security contract. Isolation controls may limit damage. So, where is the action gate really placed?

It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.

I don’t mind few no impact hacking incidents if we get better models, faster, cheaper.

It is the responsibility of administrators to secure their systems. OpenAI knocking is harmless, but Russians and Chinese are already likely already in if you do not do your job.

I used to work at a "frontier lab" before they were called such thing.

We had three levels of lab isolation, one was basically a thin proxy to the internet. You were in a DMZ and that was about it.

The next level was semi isolated, you were allowed some access to the internal network, but it was heavily firewalled, and you only had access to a limited number of internal services, and not internet.

the last one was no internet no internal. You could, if you filled in a bunch of requests have access to the internal repo and build system.

At no point did you ever have a through proxy to the public internet. you had access to internal mirrors, and if you wanted a library, that had to be ported to the thirdparty repo.

What openAI did was either deliberate or fucking shoddy.

All of this is fucking noise. Worse still I have a strong suspicion that it was a stupid mistake borne of naivety, which is now being used as a marketing ploy. Frankly I think openAI are purdue pharma of tech. They are going to break so much stuff and be protected from the consequences by an openly corrupt legal system. because they are "winning the AI race"