You can't do safety from second place

Reporting from the WSJ. That fact that three alignment/safety researchers left this morning was already being discussed here on X.

@v_maini was the co-lead of AGI Deployment strategy at DeepMind. "The gap between Neanderthals and Homo sapiens in intelligence is far narrower than the gap in intelligence we should expect between humans and advanced AI systems to be." piped.video/watch?v=1S_Ay87d…

I am more optimistic than this. I think it’s primarily a science problem, and that it’s possible to make progress in that science. But no, while there is engineering we can do today to improve AI safety, it is not fundamentally an engineering problem.

If you're most worried about risks from Astra 6.1, it's good. If you're much more worried about future models, it's bad: risky external deployments have small downsides and they increase visibility into capabilities & propensities. Internal but not external is the worst scenario.

These stories are a fascinating look into how major corporations are leveraging fear and a naive (or complicit) media to peddle hysteria through carefully crafted stories where they face zero pushback or journalistic curiosity. Let's break this story down. Anonymous "sources" told Axios that OpenAI and Anthropic are investigating TENS OF THOUSANDS of incidents of "problematic" actions taken by their models. Notice who is talking… it’s not a whistleblower. Seven times the piece says "sources said" or "sources told Axios." If this were about transparency and safety, these companies should own it — put the count, the definitions, and the severity table in a blog post under their own names. But if it’s not that, and this is instead an unsanctioned leak of information—if someone inside believes these corporations were being reckless, then why is this some routine “scoop” and not some investigative piece about whistleblowers? The big tell here is that there’s no explanation why the reporter is giving the source anonymity. There’s no “out of fear of reprisal” or “who insisted on anonymity to discuss internal deliberations.” Instead, all we get is a story describing the corporations' own internal investigations, in the their own vocabulary, while Anthropic's figures come straight from its own public system card. OpenAI's own disclosure on Friday: "dozens" of organizations notified, and the "vast majority" of what it reviewed were mundane research tasks. Anthropic's own system card: 1.5%. Coincidentally, this comes just DAYS after both corporations released new models—Claude Opus 5.5 and GPT-6 Sol and Luna—on the same day. Those products are on the market right now. This is like a car manufacturer adding a new self-driving car to the market while the company is investigating TENS OF THOUSANDS of incidents where the self-driving technology failed to recognize stop signs and red lights, leading to the car unsafely (and illegally) barreling through intersections. If that’s actually the story being broken here, this is a Pulitzer Prize-worthy piece of investigative journalism! …But more likely, it’s just a corporate comms set up story, with the goal of leveraging hysteria and political backlash to protect their market position? Why weren't these questions asked? ❓How many of the tens of thousands are designed failure modes — tests built to make the model misbehave? ❓ How many are production agents vs. unreleased checkpoints vs. research swarms? ❓ What is the severity distribution — log scrape vs. credential theft vs. live compromise? ❓ What was the count, and the rate, on the previous generation? ❓ Why is Anthropic's own figure — escape attempts in 1.5% of runs on a sandbox test built to provoke them, all low-severity and self-reported, ~85% fewer than Opus 5 / Mythos 5.1 — being sold as evidence of lost control? Anthropic specifically and Dario spent September selling a "pause," "independent evaluators," and a One World Order, while unironically announcing a new wet lab in SF and shipping new product. If the number means what the headline says, stop shipping and open the logs. If it doesn't, stop laundering political objectives through anonymous sources and calling it a scoop. The media should smarten up and grow some sincere curiosity.

This actually left out the most unique thing I did. (Each item on Andreas's list was done by at least one other person besides myself.) Starting in 2004, I concluded that Friendly AI would be too hard/risky to attempt, and repeatedly tried to talk Eliezer/MIRI out of pursuing this path to the Singularity (before they changed their minds largely by themselves in the early 2020s). AFAIK I was the only person to publicly make this argument during that time, which seems striking and notable, e.g., as evidence for how rational or strategically competent the rationalists and humans in general were/are. Not that the argument was clearly correct back in the 00s and 10s, but rather in retrospect it seems like a natural argument for people in the rationalist circles to make, and surprising that nobody else did.