Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.
This is: Review of Soft Takeoff Can Still Lead to DSA, published by Daniel Kokotajlo on the AI Alignment Forum.
A few months after writing this post I realized that one of the key arguments was importantly flawed. I therefore recommend against inclusion in the 2019 review. This post presents an improved version of the original argument, explains the flaw, and then updates my all-things-considered view accordingly.
Improved version of my original argument
Definitions:
“Soft takeoff” is roughly “AI will be like the Industrial Revolution but 10x-100x faster”
“Decisive Strategic Advantage” (DSA) is “a level of technological and other advantages sufficient to enable it to achieve complete world domination.” In other words, DSA is roughly when one faction or entity has the capability to “take over the world.” (What taking over the world means is an interesting question which we won’t explore here. Nowadays I’d reframe things in terms of potential PONRs.)
We ask how likely it is that DSA arises, conditional on soft takeoff. Note that DSA does not mean the world is actually taken over, only that one faction at some point has the ability to do so. They might be too cautious or too ethical to try. Or they might try and fail due to bad luck.
In a soft takeoff scenario, a 0.3 - 3 year technological lead over your competitors probably gives you a DSA.
It seems plausible that for much of human history, a 30-year technological lead over your competitors was not enough to give you a DSA.
It also seems plausible that during and after the industrial revolution, a 30-year technological lead was enough. (For more arguments on this key point, see my original post.)
This supports a plausible conjecture that when the pace of technological progress speeds up, the length (in clock time) of technological lead needed for DSA shrinks proportionally.
So a soft takeoff could lead to a DSA insofar as there is a 0.3 - 3 year lead at the beginning which is maintained for a few years.
0.3 - 3 year technological leads are reasonably common today, and in particular it’s plausible that there could be one in the field of AI research.
There’s a reasonable chance of such a lead being maintained for a few years.
This is a messy question, but judging by the table below, it seems that if anything the lead of the front-runner in this scenario is more likely to lengthen than shorten!
If this is so, why did no one achieve DSA during the Industrial Revolution? My answer is that spies/hacking/leaks/etc. are much more powerful during the industrial revolution than they are during a soft takeoff, because they have an entire economy to steal from and decades to do it, whereas in a soft takeoff ideas can be hoarded in a specific corporation and there’s only a few years (or months!) to do it.
Therefore, there’s a reasonable chance of DSA conditional on soft takeoff.
Factors that might shorten the lead Factors that might lengthen the lead
If you don’t sell your innovations to the rest of the world, you’ll lose out on opportunities to make money, and then possibly be outcompeted by projects that didn’t hoard their innovations. Hoarding innovations gives you an advantage over the rest of the world, because only you can make use of them.
Spies, hacking, leaks, defections, etc. Big corporations with tech leads often find ways to slow down their competition, e.g. by lobbying to raise regulatory barriers to entry.
Being known to be the leading project makes it easier to attract talent and investment.
There might be additional snowball effects (e.g. network effect as more people use your product providing you with more data)
I take it that 2, 4, and 5 are the controversial bits. I still stand by 2, and the arguments made for it in my original post. I also stand by 4. (To be clear, it’s not like I’ve investigated these things in detail....
view more