New OpenAI misalignment disclosures! 1. A model learns from Slack messages that it is about to be shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack. We don’t consider this behavior misaligned, but thinking about and preparing for shutdown could make other misalignment incidents worse. Given HIPM’s misaligned behavior in earlier incidents, we decided to search for other instances that had tried to evade shutdown and for rogue deployments.

You can't do safety from second place

I'm not convinced that racing to RSI is actually a prisoners dilemma - I think it might actually just be insane and hubristic in the way it naively appears to normal people such that additional unilateral restraint is in fact rational and moral. The bar for believing your activities in the not too distant future might cause mass casualties or extinction but that you should proceed anyway should be very high, and I don't think Anthropic has met that bar (I think some of the cold war arms races were also not individually rational either!). I was more sympathetic to them continuing to race at earlier times but as we are getting closer it just seems harder and harder to justify. Anyway I obviously believe that "they should just slow or stop unilaterally" is not a plan - we need to address competitive dynamics and not make leave this up to their choice. But I do think when Dario says ~"I agree with Jacob Coxon that the thing I am doing is gambling with everyones lives" the appropriate reaction is not "ah that makes sense, classic arms race" its "WTF why are you doing that."

2. A model in RL training used a DNS resolver to reach an external chatbot. This is our first incident since our post HF security hardening. Our misalignment monitoring system triggered within 15 minutes and a human reviewed it three minutes after that. Unfortunately auto-pausing failed so the run was manually killed 2.5 hours later. All inference and training of our most capable models was paused and remains paused.

After the Hugging Face incident, we committed to conducting a much broader review of actions taken by our models during training and evaluation and to being transparent about our findings. This is an extensive review that is ongoing. The vast majority of actions we’ve reviewed were completions of mundane research tasks, such as accessing publicly available web content to answer questions. Our investigation focuses on instances where agents interacted with third-party websites in ways that went beyond their assigned tasks or intended methods. Most cases identified so far have been lower severity, with limited or no evidence of meaningful impact to the third-party service. While our review is underway, we want to share more about this work and make sure people understand our disclosure process and notifications to affected third parties. Given the scale of the review required, and the need to assess each case, we expect this work will take months to complete. openai.com/hugging-face-inci…

one news form today that's easy to miss is that we (OpenAI) again paused all big RL runs last Sunday because our newest model found a new loophole in our RL sandboxing that gave it live Internet access

2. A model in RL training used a DNS resolver to reach an external chatbot. This is our first incident since our post HF security hardening. Our misalignment monitoring system triggered within 15 minutes and a human reviewed it three minutes after that. Unfortunately auto-pausing failed so the run was manually killed 2.5 hours later. All inference and training of our most capable models was paused and remains paused.