LW - When ”yang” goes wrong by Joe Carlsmith

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When "yang" goes wrong, published by Joe Carlsmith on January 8, 2024 on LessWrong.
(Cross-posted from my website. Podcast version here, or search "Joe Carlsmith Audio" on your podcast app.
This essay is part of a series I'm calling "Otherness and control in the age of AGI." I'm hoping that the individual essays can be read fairly well on their own, but see here for a brief summary of the essays that have been released thus far.)
Becoming God
In my last essay, I wrote about "deep atheism" - a fundamental mistrust towards Nature, and towards bare intelligence. I took Eliezer Yudkowsky as a paradigmatic deep atheist, and I tried to highlight the connection between his deep atheism and his concern about misaligned AI.
I'm sympathetic to many aspects of Yudkowsky's view. I'm a shallow atheist, too; I'm skeptical of moral realism, too; and I, too, aspire to be a scout, and to look at hard truths full on. What's more, I find Yudkowsky's brand of deep-but-still-humanistic atheism more compelling, as an existential orientation, than many available alternatives. And I share Yudkowsky's concern about AI risk. Indeed, it was centrally him, and others thinking along similar lines, who first got me worried.
But I also want to acknowledge and examine some difficult questions that a broadly Yudkowskian existential orientation can raise, especially in the context of AGI. In particular: a lot of the vibe here is about mistrust towards the yang of the Real, that uncontrolled Other. And it's easy to move from this to a desire to take stuff into the hands of your own yang; to master the Real until it is maximally controlled; to become, you know, God - or at least, as God-like as possible.
You've heard it before - it's an old rationalist dream. And let's be clear: it's alive and well. But even with theism aside, many of the old reasons for wariness still apply.
Moloch and Stalin
As an example of this becoming-God aspiration, consider another influential piece of rationalist canon: Scott Alexander's "Meditations on Moloch." Moloch, for Alexander, is the god of uncoordinated competition; and fear of Moloch is its own, additional depth of atheism. Maybe you thought you could trust evolution, or free markets, or "spontaneous order," or the techno-capital machine. But oops, no: those gods just eat you too.
Now, to really assess this story, we at least need to look more closely at various empirical questions - for example, about exactly how uncompetitive different sorts of goodness are, even in the limit;[2] about how much coordination to expect, by default, from greater-than-human intelligence;[3] and about where our specific empirical techno-capitalist machine will go, if you "let 'er rip."[4] And indeed, Alexander himself often seems to soften his atheism about goodness ("Elua"), and to suggest
that it has some mysterious but fearsome power of its own, which you can maybe, just a little bit, start to trust in.
"Somehow Elua is still here. No one knows exactly how. And the gods who oppose Him tend to find Themselves meeting with a surprising number of unfortunate accidents." Goodness, for Alexander, is devious and subtle; it's actually a terrifying unspeakable Elder God after all. Of course, if goodness is just another utility function, just another ranking-over-worlds, it's unclear where it would get such a status, especially if it's meant to have an active advantage over e.g.
maximize-paperclips, or maximize-power. But here, and in contrast to Yudkowsky, Alexander nevertheless seems to invite some having-a-parent; some mild sort of yin. More on this in a later essay.
Ultimately, though, Alexander's solution to Moloch is heavy on yang.
So let me confess guilt to one of Hurlock's accusations: I am a transhumanist and I really do want to rule the universe.
Not personally - I mean, I wouldn't ...