LW - Apocalypse insurance, and the hardline libertarian take on AI risk by So8res

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Apocalypse insurance, and the hardline libertarian take on AI risk, published by So8res on November 28, 2023 on LessWrong.
Short version: In a saner world, AI labs would have to purchase some sort of "apocalypse insurance", with premiums dependent on their behavior in ways that make reckless behavior monetarily infeasible. I don't expect the Earth to implement such a policy, but it seems worth saying the correct answer aloud anyway.
Background
Is
advocating for AI shutdown contrary to
libertarianism? Is advocating for AI shutdown like arguing for markets that are free except when I'm personally uncomfortable about the solution?
Consider the old adage "your right to swing your fists ends where my nose begins". Does a libertarian who wishes not to be punched, need to add an asterisk to their libertarianism, because they sometimes wish to restrict their neighbor's ability to swing their fists?
Not necessarily! There are many theoretical methods available to the staunch libertarian who wants to avoid getting punched in the face, that don't require large state governments. For instance: they might believe in private security and arbitration.
This sort of thing can get messy in practice, though. Suppose that your neighbor sets up a factory that's producing quite a lot of lead dust that threatens your child's health. Now are you supposed to infringe upon their right to run a factory? Are you hiring mercenaries to shut down the factory by force, and then more mercenaries to overcome their counter-mercenaries?
A staunch libertarian can come to many different answers to this question. A common one is: "internalize the externalities".[1] Your neighbor shouldn't be able to fill your air with a bunch of lead dust unless they can pay appropriately for the damages.
(And, if the damages are in fact extraordinarily high, and you manage to bill them appropriately, then this will probably serve as a remarkably good incentive for finding some other metal to work with, or some way to contain the spread of the lead dust. Greed is a powerful force, when harnessed.)
Now, there are plenty of questions about how to determine the size of the damages, and how to make sure that people pay the bills for the damages they cause. There are solutions that sound more state-like, and solutions that sound more like private social contracts and private enforcement. And I think it's worth considering that there are lots of costs that aren't worth billing for, because the cost of the infrastructure to bill for them isn't worth the bureaucracy and the chilling effect.
But we can hopefully all agree that noticing some big externality and wanting it internalized is not in contradiction with a general libertarian worldview.
Liability insurance
Limited liability is a risk subsidy. Liability insurance would align incentives better.
In a saner world, we'd bill people when they cause a huge negative externality (such as an oil spill), and use that money to reverse the damages.
But what if someone causes more damage than they have money? Then society at large gets injured.
To prevent this, we have insurance. Roughly, a hundred people each of whom have a 1% risk of causing damage 10x greater than their ability to pay, can all agree (in advance) to pool their money towards the unlucky few among them, thereby allowing the broad class to take risks that none could afford individually (to the benefit of all; trade is a positive-sum game, etc.).
In a sane world, we wouldn't let our neighbors take substantive risks with our lives or property (in ways they aren't equipped to pay for), for the same reason that we don't let them steal. Letting someone take massive risks, where they reap the gains (if successful) and we pay the penalties (if not), is just theft with extra steps, and society should treat it as such. The freedo...