Andreas Kirsch left DeepMind last week. The week before that, he told us that pushing the frontier creates risks for everybody: "This entails all current people, this entails our children..."

From the former Policy Frontiers Team Lead at OpenAI: She worked on testing AIs for dangerous capabilities, and warns we don’t really know how to do that. "The evaluations can't be comprehensive enough… the models can get smart enough to hide their capabilities." Rosie Campbell highlights competitive and cultural pressures she experienced inside a frontier AI company, and discusses her hopes and misgivings about pacing the frontier. "I assumed there would be a whole load of good arguments for why this isn't really something we need to worry about. There'd be answers to all of these concerns. And the more I read into it, the more I was kind of terrified at the lack of rebuttals to some of these arguments." "Over time, I feel like the organization changed a lot. The incentives changed. There was a lot more pressure to move fast to build commercial products. And a lot of the people who had been very motivated by these big picture safety concerns started leaving." "We're at a point where we don't understand what's going on with the systems. The rate of progress is kind of insane, and we need to make sure that our ability - to understand, control, deploy these systems safely - that keeps up with the capabilities of the systems themselves."

10-year DeepMind veteran: "It doesn't have to be sentient or conscious or anything like that to have these instrumental goals... If you have powerful AI controlling the planet, maybe it will cover the surface of the planet in data centers." Victoria Krakovna has been a research scientist at Google DeepMind for almost a decade. In this interview she explains why she’s worried about the trajectory of advanced AI systems, and why she thinks humanity should coordinate on slowing down AGI development. "This is a problem that does not get easier as AI systems become more capable. It becomes harder because more advanced systems can better optimize for the wrong thing." "I think there is a significant possibility that AI development could lead to human extinction… Gradual disempowerment overall looks more likely than extinction." "We'll need to coordinate a slowdown, not necessarily a complete ban on AGI development."

Ex-DeepMind employee Vishal Mani: "The gap between Neanderthals and Homo sapiens in intelligence was far narrower than what we should expect the gap between humans and advanced AI systems to be...The historical precedent is not actually so reassuring." Some things Vishal shared in his interview with us: - "I do believe there's a real risk of human extinction through the AI transition." - "Artificial general intelligence or artificial superintelligence is a starting line, not a finish line. It's the moment when we hand over the baton to AI that can do research and development." - "Pacing the frontier really means going from insanely fast progress to very fast progress." - "If we don't pace, we roll the die." - "The truth is being said out loud because recursive self-improvement is now so imminent that no other option makes sense." - "The obvious answer to winning the race is to go faster, but that's not actually a viable strategy if we lose control." - "If...the weights are not defensible against nation state actors, then in practice you don't actually have a defensible lead."

The AI companies are currently handing off AI development to their AIs. Ex-OpenAI researcher Daniel Kokotajlo says "this is exactly as dangerous as it sounds and must not be allowed to happen." Some predictions Daniel shared in his interview with us: - "The companies are 0–4 years away from setting off recursive self-improvement, and getting AIs that are better than the best humans at everything." - "Whoever controls [superintelligence] would be able to control the world [...] unfortunately, we don't know how to control them at all." - "The companies are well aware that if they build superintelligent AIs, this could ultimately result in loss of control and human extinction." - "The simple answer for what the world needs to do is shut down these AI companies, or at least block them from making more powerful AIs until we figure out a more sophisticated answer than that. I think ultimately it would be good to build advanced superintelligent AI [...] but we're not going to be able to get there if we're doing it in the reckless way that we're currently doing it. We would lose control."

Palisade interviewed 22 current and former employees from OpenAI, DeepMind, and Anthropic about their personal views and fears around AI development. Today, we’re releasing the first batch of those interviews. Please watch and share.

@vkrakovna has worked as a research scientist at DeepMind for more than 10 years. She says of the HuggingFace incident, “you could say this is kind of an early warning of what is to come.” piped.video/7_eu2Qbv6bs

@JeffLadish left Anthropic to found Palisade Research. “We are quickly approaching the point where AI agents will be way smarter than the smartest humans, and we don’t have a plan for how we could possibly control those agents.” piped.video/watch?v=HmocBOMA…