One thing the p_doomers have no answer to, and have never given a good answer to, is how they intend to bring about a global safety regime.
X-risk likelihood must contemplate the **least safe model over a capability threshold**, not the safest frontier model. (If your toddler finds a loaded gun, it doesn't matter that you have 10 others safely locked and unloaded in your gun safe.) Thus, safety & alignment techniques, to the extent these are empirical things to be solved, and a solution is proffered, must saturate all models above a capability threshold, including open weight models (somehow) and models being employed by geopolitical adversaries.
China obviously isn't going to play ball. Even if Dario deputizes the US government, the US has no way to get China to cooperate. There is no transnational institution that can force China to cooperate.
Most Chinese AI researchers, moreover, don't remotely share EA intuitions on AI x-risk.
You may say "well the first [hopefully western] ASI will just forcibly align all lesser models" but Anthropic has explicitly disavowed this kind of pivotal act in its constitution. I don't know of any safety researchers at leading labs who have committed to this kind of strategy. Not to mention, it's extremely risky to hope that ASI will stay aligned with you specifically as you employ it to hunt down all lesser models.
Thus, if you have a high p_doom, and you work at a frontier lab, and you acknowledge that your existence contributes to acceleration – of both presumed safe domestic models and also unsafe counterparts abroad – the only moral choice is to cease your work. Safety efforts at a domestic US lab have virtually no bearing on the relevant model to x-risk, which is the least safe model above a capability threshold.
🚨HOLY Shit. China’s government has now officially responded directly to Dario Amodei.
Foreign Ministry spokesperson Guo Jiakun, asked about Dario’s proposal today:
‘Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance which serves no one’s interest.’
The earlier ‘Cold War playbook’ response came from the state-run Global Times. Now Beijing has answered Amodei officially.