Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Rocket Alignment Problem, Part 2, published by Zvi on May 1, 2023 on LessWrong.
Previously (Eliezer Yudkowsky): The Rocket Alignment Problem.
Recently we had a failure to launch, and a failure to communicate around that failure to launch. This post explores that failure to communicate, and the attempted message.
Some Basic Facts about the Failed Launch
Elon Musk’s SpaceX launched a rocket. Unfortunately, the rocket blew up, and failed to reach orbit. SpaceX will need to try again, once the launch pad is repaired.
There was various property damage, but from what I have seen no one was hurt.
I’ve heard people say the whole launch was a sshow and the grounding was ‘well earned.’ How the things that went wrong were absurd, SpaceX is the worst, and so on.
The government response? SpaceX Starship Grounded Indefinitely By FAA.
An FAA spokesperson told FLYING that mishap investigations, which are standard in cases such as this, “might conclude in a matter of weeks,” but more complex investigations “might take several months.”
Perhaps this will be a standard investigation, and several months later everything will be fine. Perhaps it won’t be, and SpaceX will never fly again because those in power dislike Elon Musk and want to seize this opportunity.
There are also many who would be happy that humans won’t get to go into space, if in exchange we get to make Elon Musk suffer, perhaps including those with power. Other signs point to the relationships with regulators remaining strong, yet in the wake of the explosion the future of Starship is for now out of SpaceX’s hands.
A Failure to Communicate
In light of these developments, before we knew the magnitude or duration of the grounding, Eliezer wrote the following, which very much failed in its communication.
If the first prototype of your most powerful rocket ever doesn’t make it perfectly to orbit and land safely after, you may be a great rocket company CEO but you’re not qualified to run an AGI company.
(Neither is any other human. Shut it down.)
Eliezer has been using the rocket metaphor for AI alignment for a while, see The Rocket Alignment Problem.
I knew instantly both what the true and important point was here, and also the way in which most people would misunderstand.
The idea is that in order to solve AGI alignment, you need to get it right on the first try. If you create an AGI and fail at its alignment, you do not get to scrap the experiment, learn from what happened. You do not get to try, try again until you succeed, the way we do for things like rocket launches.
That is because you created an unaligned AGI. Which kills you.
Eliezer’s point here was to say that the equivalent difficulty level and problem configuration to aligning an AGI successfully would be if Musk stuck the landing on Starship on the first try. His first attempt to launch a rocket would need to end up safely back on the launching pad.
The problem is that the rocket blowing up need not even get one person killed, let alone kill everyone. The rocket blowing up caused a bunch of property damage. Why Play in Hard Mode (or Impossible Mode) when you only need to Play in Easy Mode?
Here were two smart people pointing out exactly this issue.
Jeffrey Ladish: I like the rocket analogy but in this case I don’t think it holds since Elon’s plans didn’t depend on getting it right the first try. With rockets, unlike AGI, it’s okay to fail first try because you can learn (I agree that Elon isn’t qualified to run an AGI company)
Eliezer: Okay if J Ladish didn’t get it, this was probably too hard to follow reliably.
The analogy is valid because Elon would’ve preferred to stick the landing first try, and wasn’t setting up deliberately to fail where Starship failed. If he had the power to build a non-omnicidal superintelligence on his first try, he could’ve also ...
view more