Thinking more about this, my moral intuitions here:
1. I take as a given (and I imagine you do too) that a future competitor's version of this technology (which, say, lacks guardrails and disclosures) will absolutely be used to great harm, either overtly (scams) or subtly (rising alienation—not clear if guardrails/disclosure even matter there).
2. Consequentialistically: if I create and celebrate the first instance of such a technology, I signal that it's possible and show its value, encouraging others to develop rivals—four minute mile. It's hard to imagine that I won't accelerate time-to-first-misuse.
3. Deontologically: sure, probably "if I don't do it someone else will", but I would not want to create the first instance of a technology if I already know later iterations of it will surely be seriously misused.
4. Virtue-ethically: I believe that knowingly abetting a technology which others will later seriously misuse is corrosive to the soul.
5. Sociologically: I think that if I signal "this kind of technology is good to make (with the right safeguards)", I encourage the industry's worst excesses. I don't want to normalize the creation of negative social and practical externalities. At the very least, I'd want to signal that it is the duty of such a project to deeply and openly grapple with the weight of the box it's opening.
6. Democratically: if I created a technology like this, I believe I'd be continuing our industry's history of imposing significant indirect taxes on the public without representation. Much of the public resents our industry in part because they feel as if technological upheaval is something that's being "done to them", without their consent or desire. A technology this weighty should require democratic due process.
7. Practically: is it worth it? Less uncanny valley in BDR training, and in role-playing games? Man, this just seems like an awful lot of commons to burn to boil a pint of water.