I saw @tylercowen while walking to get icecream on the phone with my mom and was so surprised I blurted out ‘Hi Tyler’ and he said ‘Hi!’ and then my mom was like ‘oh is that I friend of yours’ and I was like no mom much worse, that is a famous intellectual who doesn’t know me and I got too excited - and she said ‘well he sounded nice!’

‘The first time I got a mouse, I went to look at Yellowstone. I can’t touch grass. you can’ man what a bop

I love the start of a new google doc. The flicking cursor is like a little heart beat or a butterfly wing

David's critique is substantive and deserves careful consideration. My impression of David from our time overlapping at OpenAI is of a sober and thoughtful person who didn't approach the work from an ideological lens. This isn't someone who came in with pre-formed opinions about existential risk; he wasn't to my knowledge an EA. He was consistently productive, collaborative, and well-reasoned in his approach to safety. The core of his critique - that professional safety engineering practices from other fields haven't taken enough root in AI safety - is substantial and correct. We're entering a phase shift where AI safety organizational practices that worked a year ago, for models of 2025 capability levels, are not sufficient to prevent serious incidents. This is a new and different world and every organization needs to uplevel accordingly. I have much more confidence in my colleagues at OpenAI than David expresses in this article, and OpenAI does much more on this front than it gets credit for. But the actual bar for OpenAI as a whole isn't just "is it a clear field leader on alignment work, and is it successfully addressing issues as they arise?" - which is no small bar to start with - but rather, "can it earn the full trust and confidence of the public that it can safely pursue a path to superintelligence?" Because if it doesn't clear that bar, it will lose the license to operate. (Given that the public is actively debating whether to explicitly ban superintelligence and/or recursive self-improvement, I don't think this is an exaggeration.) Or, much worse, it could have a critical safety incident where the real harm is unacceptable. That bar can only be hit by increasing the level of high-reliability safety engineering practices, embracing extreme and proactive candor around incident disclosure, and implementing third party verification that is unimpeachable from a conflicts perspective and persuasive to credible experts. (And I think it's even worth it to persuade the hostile ones.)

im working on something im really excited about

Quite the essay. “Perhaps I should have stayed and fought for fundamental shifts in our staffing and culture, but in practice, my colleagues and I were so busy sprinting that we seldom had the chance to consider big changes, much less to actually make them.”

I hope one day people joke about AI existential risk the way they joke about Y2K

This is pretty cool and punk

This one seems worth listening to. I've found Joe Lonsdale to be among the most persuasive voices for the broadly AI regulation skeptical perspective, even if I strongly disagree. I appreciate that he acknowledges the risks are real even if he disagrees what to do about it.

We need bigger venues for our salons, over 500 people want to come tonight

I met someone who is in town for the curve and I asked them what they wanted to get out of it. Their plan seemed to be focused on others changing their mind about a topic. I told them this was a bad use of their time and they should consider using the curve to see how others can help ~them. I was like there is prob a specific reason why you in particular are attending. Figure out what that reason might be and what you want other people’s help on. Come with some ideas and use other people to sharpen those ideas. I think people should be way more self centric in their conference purists. If you know what you want from a conference, or the world, it is easy to ask people to help you get it - or for them to tell you the way you are thinking is slightly wrong. If you go to a conference for any reason other than to vibe with your friends (which is a valid way to conference), have it be to learn all the ways your assumptions might be wrong

Another incentive alignment I would like to see is shared recruiters for AI safety, where the recruiters get paid % share / placement fees. Good recruiters are bottlenecks right now

My friends just walked pass my apartment and yelled my name from the street. I find this so platonically romantic

please stop saying that airplanes can "fly". they are only exploiting aerodynamic regularities to produce flight-like behavior

I have recently shifted from saying 'ai risk' to be more precise and say 'ai danger' and 'x-risk'. I think ai risk as a blanket term (much like ai safety) is too soft and large

one of the reasons i was down to organize manifest was austin trying one of these so called 'weirder financial arrangements that turn out to be the right tool for a specific job'. he basically was like i will pay you a contractor rate, a profit share of this years conference, and a smaller profit share of all future years to align the incentives. and i was like that's so weird/cool im in

i guess i assumed this hearing would be covered more so i didn't bother to tweet much concrete about it, but apparently not many people even on the TL have the stomach to watch two hours of congress. this was a hearing before the homeland security committee on *rogue ai* specifically. as far as i could tell it was extremely bipartisan. @HawleyMO , the senator in the middle here who brought out the huggingface slide, is a hyper-conservative missouri senator. the people testifying were @ChrisPainterYup (METR president), @DKokotajlo (ai2027/2040), @MariusHobbhahn (apollo research CEO), and a cybersecurity expert and legal expert i don't know. so... pretty fucking stacked on testimony not every senator asked good questions. but most of them did. all of them very clearly already knew plenty of details about the huggingface incident and multiple other incidents. most of them had clear understanding of terms like "misalignment", "recursive self improvement", "chain of thought / chain of thought monitoring", etc etc!! they all clearly had their own policy angles they liked and were pushing, implicitly or explicitly. but as best as i could tell: - it seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. they independently brought up how bad it would be for rogue AI agents to move laterally between data centers - they all seemed to basically take RSI quite seriously. not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much much more capable, much much less controllable, and causing much more damage or loss of life. - they mostly seemed to have a clear intuitive understanding of why RSI might lead to misalignment. it didn't take much, it was a really simple chain of reasoning they themselves laid out, "if the models right now are kinda misaligned and we don't know what they're doing sometimes, and then we have them build the next models and those ones build the next ones and so on, and we're having to ask the AI's what's going on to even understand it with how fast it's going, we really won't know how they're built or what they'll do" - at one point a senator said flat out "should we just make RSI illegal?" (not a joke! this really happened!) - every single senator seemed to think it was obvious we needed *both* much harsher liability regimes for ai developers and also new legislation, both very quickly. this was the complete consensus, difference basically just being degree. - they were largely quite concerned about china, and falling behind china. but this clearly wasn't the be-all end-all. as mentioned above they all thought it was obvious necessary to stop rogue ai even if it meant moving more slowly. - at one point a senator said "china is a tightly controlled communist society, they're going to run into these same issues, and there's absolutely no way they're just going to let them run wild, they'll obviously stop at that point, so we're not really in a race" - on the other hand another senator said "china isn't concerned with human life"... dario-modeing i came away from this incredibly encouraged. i don't know exactly what's going to happen here, and ofc this is a small subset of congress and one hearing, and they each have their own policy agendas most of which are probably super divergent from mine. but holy shit !!! they understood a lot of what was going on! they care!! this is an obviously salient political and safety issue to them, and clearly bipartisan! the US government is awake.

Mosquitoes are the deadliest animals to humans on earth, killing approximately 1 million people every year. Living with them is a choice. We have the tech to eliminate entire species of mosquitoes in populous areas. We should do it. This EO gets us one step closer to that goal.

I am recently reminded that some people consider having a strategy and then executing it to be deeply suspicious behavior, most particularly when it succeeds.

Hosting a salon with Convergent Research tomorrow! we are way way over subscribed but shoot your shot if we're mutuals luma.com/foresight-lqwp