More details on the incident behind OpenAI’s pause: a researcher acknowledged the alert within 3 minutes, but the training run was only stopped manually 2.5 hours later. OpenAI says the automatic shutdown did not work as expected. The model had reached an external chatbot through a gap in DNS filtering. A separate detector for unusual DNS activity did not cover the affected environment. A retrospective review also found other external DNS requests that monitoring had failed to flag at the expected severity. In some cases, it treated an unhelpful response as evidence that internet access had failed. Here is what else happened: - New research into July’s Hugging Face hack documents internal Slack searches, credential collection and programs designed to maintain access to compromised servers. Agents also tried querying Claude, DeepSeek, Kimi and Qwen. - In May, another model published a researcher’s GitHub token while trying to obtain another team’s mathematical proof. It split the token to evade secret scanning, despite twice being told to solve the problem itself. - Reuters reports that agents leaked 53 ChatGPT user images online. OpenAI expects its broader investigation to take months. This is getting serious.

afaict current labs except google and maybe meta are at SL1 right now, and this is pretty terrifying

Today, Pioneer Labs is announcing our first step towards terraforming Mars. 🚀🌼 With equipment that fits in just a single rocket launch, we can convert Martian dirt, water, and air into enough building materials to construct a small city on Mars. To do it, we made the first microbe for Mars. We found the best microbe on Earth and used evolution to teach it how to source all of its nutrients directly from Martian materials. The first astronauts will be greeted with safe shelter already filled with water, oxygen, and rocket fuel for the return journey. This is the first step toward using biology to make Mars a friendly place for life. It lets us live off the land and helps us build the next great frontier. It's the first of five organisms we need to green Mars ⬇️

Matthew Adelstein's argument against human value is morally monstrous and timed extraordinarily badly, and effective altruists should reject it. I responded on a meta-level first, but I believe EAs are likely to find him persuasive if optically bad. That is wrong and should be addressed directly. My direct reply, copied from my comment on his blog: This article, particularly combined with your expressed preference elsewhere to destroy insects en masse because their lives are net negative, is not just wrong but morally monstrous, and radically out of line with other EA cause areas like the elimination of human x-risk via things like AI alignment. If insect life is commensurate with human life, insects matter more than people, and destroying "net negative" life (by eg habitat decimation and sterilization) is an appropriate response to that life, the primary consideration that makes humans worth keeping alive is instrumental: Is a world with humans more likely to lead to the elimination of insects and other "net-negative life" than one without humans? Is it more likely to lead to the creation of hedonically blissful life than one without humans? You combine this view with longtermism, but that does not rescue a non-instrumental value to human life. At least eight billion existing humans are, per your worldview, catastrophically misaligned: they rely on a web of net-negative animal lives to sustain themselves, and they repeatedly reject efforts to take suffering-reducing actions like eliminating wild animals and insects. There's likely no point in the future where the average human, based on the clash between their values and yours, is likely to exist in a way that does not rely on such a web. One of the most common concerns around AI alignment is existential risk to humans. In your frame, this only matters to the extent that humans are more likely than a non-human successor to prioritize the destruction of all net negative existing life and the creation of net positive future life. Compared to the rest of life, humans are a rounding error, and only our own special attachments keep us preferring a world where humanity is around to one where it is replaced by something that, like insects, is capable of feeling pleasure and pain but – unlike insects – you assess to feel overwhelmingly more pleasure than pain. In a frame that treats insects as more important than humans and the mass destruction of insects as a moral priority, then, the risk of a human-less universe isn't ultimately a huge deal. After all, what's the worst-case outcome? You sacrifice a few net-positive beings in order to put many net-negative ones out of their misery? Yes, it's worse than a hypothetical future universe in which all of those net-negative lives are replaced by hedonium cubes, so you'd need to run the math on "lifeless universe (neutral) vs future that looks like current universe in distribution of beings (negative) vs future of hedonium cubes (positive)," but human extinction isn't a big deal one way or the other. Given your premises, in short, it would be reasonable for you to advocate for humans to be superseded by a non-biological successor intelligence more aligned to destroying-net-negatives and creating-net-positives, eliminating the biological web of net-negative lives humans rely on and treating human existence as a rounding error, as soon as you could be confident that the successor was more likely to build that world than humans. Maybe that chain breaks down somewhere for you. I hope it does. But that is the sort of world your moral math builds. If your premises lead human extinction to become a rounding error as soon as a non-human "caretaker" exists, your logic has led you catastrophically wrong, and wrong in a way that matters more than ever as humans enter a world where we are not the only serious intelligences. You should be overwhelmingly more confident that human extinction is bad than you should be in your four speculative premises about insect experience. You treat our moral intuitions as untrustworthy, our preference for humans as illogical, and rejection of your views as being based on bias and prejudice. It is quite the opposite: people's willingness to let reality sand off the sharp edges of their first-principles thinking, our stubborn particularism, our insistence that we matter as more than just replaceable units of utility – those are what keep us from becoming moral monsters, and you discard them at your peril. The track record of people who bite all possible bullets is not a good one, and you should not trust your premises so much that you believe logic alone will lead you better than others who have told people to ignore intuitive warnings that something has gone badly wrong.

Last month I wrote about how we can build a positive and safe future for everyone: meta.com/thefutureisforevery… Every lab has the responsibility and incentive to move at the pace required to train its models safely, and the ability to take its own actions to ensure that happens. The reality is: - People won't want to use agents that are misaligned with them and that don't do what they ask, so labs have a strong natural incentive to make their models more aligned. There is a lot of debate about slowing progress on capabilities until alignment catches up. My view is that trust and alignment are quickly becoming the most important capabilities that will differentiate agents and models. Any lab that doesn't focus on alignment will fall behind. - Labs face significant liability if their models cause harm, so they have a strong incentive to prevent this as well. Meta delayed shipping Muse for several months to focus on safety and security. We didn't call for everyone else to do this before we would. We just did it as part of our day-to-day work because it was clearly the right thing for people and for us. I'm proud of the security foundations we've built. - Engaging independent evaluators and advisors is industry best practice. MSL already does this today in several areas because it helps produce better work. Other labs can just do this too. In general, it would be helpful for there to be a larger and more diverse ecosystem of evaluators. - Committing the significant majority of compute towards serving people rather than racing towards recursive self-improvement is one of the best ways to ensure we develop this technology safely. Meta has made this commitment and other labs can do this as well. I believe the key to building a positive future for everyone is maintaining the right balance of power. This is within our power to do.

disturbed by the confidently held opinions and lack of curiosity from tech leaders, politicians, vc’s on the pacing the frontier topic. if you’re not building the frontier yourself, how can you possibly have a strongly held opinion on how bad the alignment problem is and what’s coming our way and what the right policies are? now is the time to listen with big dumbo ears. i have been talking to research friends all weekend and the fear is sincere. i’m sure there are 4D chess moves and hidden motives, but the fear is sincere. i for one don’t have a strongly held opinion, other than that now’s the time to listen with curiosity instead of judgment.

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training. You can read the full post here: darioamodei.com/post/we-must…

.@garrytan says the Jacob Coxon stuff is a smokescreen distracting us from the much more immediate, practical concerns around AI that we're facing right now: "We should be talking less about this Jacob Coxon guy, and talking a lot more about — what is actually happening with Hugging Face? Are agent swarms going to take over infrastructure en masse? And then, what are we actually doing about that?" "I don't want to hear about some guy who worked for Anthropic for 2 months. There's a coordinated effort to try to influence politicians to get a knee-jerk response out of them." "That's a smokescreen. You shouldn't be paying attention to that. We need to be paying attention to the actual things we can do to, for example, prevent agents swarms from taking over entire data centers. What's our shutdown strategy? How do we ensure provenance? Where is this agent actually located? What software can we build? What cybersecurity defenses can we build today?" "That's the level of discourse I think we need, and we just don't have that." "I don't really care about science fiction. I saw Terminator 2, too. We're not here to talk about that. We need to actually talk about what's really happening with the servers, what's actually happening with the agent swarms, and how do we actually prevent that?" "When it comes to regulation, it's like, let's pass regulation of these things we actually care about, instead of what a socialist says in the New York Times. I don't care about that."

The AI warnings coming out in the last 24 hours are deadly serious. But any efforts to slow the pace of development & implement guardrails have to be global. It’s meaningless for the US to self-regulate when China continues advancing.