LW - AI, centralization, and the One Ring by owencb

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI, centralization, and the One Ring, published by owencb on September 13, 2024 on LessWrong.
People thinking about the future of AI sometimes talk about a single project 'getting there first' - achieving AGI, and leveraging this into a decisive strategic advantage over the rest of the world.
I claim we should be worried about this scenario. That doesn't necessarily mean we should try to stop it. Maybe it's inevitable; or maybe it's the best available option. But I think that there are some pretty serious reasons for concern. At minimum, it seems important to stay in touch with those.
In some ways, I think a single successful AGI project would be analogous to the creation of the One Ring. In The Lord of the Rings, Sauron had forged the One Ring, an artifact powerful enough to gain control of the rest of the world. While he was stopped, the Ring itself continued to serve as a source of temptation and corruption to those who would wield its power.
Similarly, a centralized AGI project might gain enormous power relative to the rest of the world; I think we should worry about the corrupting effects of this kind of power.
Forging the One Ring was evil
Of course, in the story we are told that the Enemy made the Ring, and that he was going to use it for evil ends; and so of course it was evil. But I don't think that's the whole reason that forging the Ring was bad.
I think there's something which common-sense morality might term evil about a project which accumulates enough power to take over the world. No matter its intentions, it is deeply and perhaps abruptly disempowering to the rest of the world. All the other actors - countries, organizations, and individuals - have the rug pulled out from under them. Now, depending on what is done with the power, many of those actors may end up happy about it.
But there would still, I believe, be something illegitimate/bad about this process. So there are reasons to refrain from it[1].
In contrast, I think there is something deeply legitimate about sharing your values in a cooperative way and hoping to get others on board with that. And by the standards of our society, it is also legitimate to just accumulate money by selling goods or services to others, in order that your values get a larger slice of the pie.
What if the AGI project is not run by a single company or even a single country, but by a large international coalition of nations? I think that this is better, but may still be tarred with some illegitimacy, if it doesn't have proper buy-in (and ideally oversight) from the citizenry. And buy-in from the citizenry seems hard to get if this is occurring early in a fast AI takeoff. Perhaps it is more plausible in a slow takeoff, or far enough through that the process itself could be helped by AI.
Of course, people may have tough decisions to make, and elements of illegitimacy may not be reason enough to refrain from a path. But they're at least worth attending to.
The difficulty of using the One Ring for good
In The Lord of the Rings, there is a recurring idea that attempts to use the One Ring for good would become twisted, and ultimately serve evil. Here the narrative is that the Ring itself would exert influence, and being an object of evil, that would further evil.
I wouldn't take this narrative too literally. I think powerful AI could be used to do a tremendous amount of good, and there is nothing inherent in the technology which will make its applications evil.
Again, though, I am wary of having the power too centralized. If one centralized organization controls the One Ring, then everyone else lives at their sufferance. This may be bad, even if that organization acts in benevolent ways - just as it is bad for someone to be a slave, even with a benevolent master[2].
Similarly, if the state is too strong relative to its citize...