LW - An AI Manhattan Project is Not Inevitable by Maxwell Tabarrok

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An AI Manhattan Project is Not Inevitable, published by Maxwell Tabarrok on July 7, 2024 on LessWrong.
Early last month, Leopold Aschenbrenner released a
long essay and
podcast outlining his projections for the future of AI. Both of these sources are full of interesting arguments and evidence, for a comprehensive summary see
Zvi's post here. Rather than going point by point I will instead accept the major premises of Leopold's essay but contest some of his conclusions.
So what are the major premises of his piece?
1. There will be several orders of magnitude increase in investment into AI. 100x more spending, 100x more compute, 100x more efficient algorithms, and an order of magnitude or two gains from some form of "learning by doing" or "unhobbling" on top.
2. This investment scale up will be sufficient to achieve AGI. This means the models on the other side of the predicted compute scale up will be able to automate all cognitive jobs with vast scale and speed.
3. These capabilities will be essential to international military competition.
All of these premises are believable to me and well-argued for in Leopold's piece.
Leopold contends that these premises imply that the national security state will take over AI research and the major data centers, locking down national secrets in a race against China, akin to the Manhattan project.
Ultimately, my main claim here is descriptive: whether we like it or not, superintelligence won't look like an SF startup, and in some way will be primarily in the domain of national security.
By late 26/27/28 … the core AGI research team (a few hundred researchers) will move to a secure location; the trillion-dollar cluster will be built in record-speed; The Project will be on.
The main problem is that Leopold's premises can be applied to conclude that other technologies will also inevitably lead to a Manhattan project, but these projects never arrived. Consider electricity. It's an incredibly powerful technology with rapid scale up, sufficient to empower those who have it far beyond those who don't and it is essential to military competition. Every tank and missile and all the tech to manufacture them relies on electricity.
But there was never a Manhattan project for this technology. It's initial invention and spread was private and decentralized. The current sources of production and use are mostly private.
This is true of most other technologies with military uses: explosives, steel, computing, the internet, etc. All of these technologies are essential in the government's monopoly on violence and it's ability to exert power over other nations and prevent coups from internal actors. But the government remains a mere customer of these technologies and often not even the largest one.
Why is this? Large scale nationalization is costly and unnecessary for maintaining national secrets and technological superiority. Electricity and jet engines are essential for B-2 bombers, but if you don't have the particular engineers and blueprints, you can't build it. So, the government doesn't need to worry about locking down the secrets of electricity production and sending all of the engineers to Los Alamos.
They can keep the first several steps of the production process completely open and mix the outputs with a final few steps that are easier to keep secret.
To be clear, I am confident that governments and militaries will be extremely interested in AI. They will be important customers for many AI firms, they will create internal AI tools, and AI will become an important input into every major military. But this does not mean that most or all of the AI supply chain, from semi-conductors to data-centers to AI research, must be controlled by governments.
Nuclear weapons are outliers among weapons technology in terms of the proportion of the supply chai...