LW - What mistakes has the AI safety movement made? by EuanMcLean

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What mistakes has the AI safety movement made?, published by EuanMcLean on May 24, 2024 on LessWrong.
This is the third of three posts summarizing what I learned when I interviewed 17 AI safety experts about their "big picture" of the existential AI risk landscape: how AGI will play out, how things might go wrong, and what the AI safety community should be doing. See here for a list of the participants and the standardized list of questions I asked.
This post summarizes the responses I received from asking "Are there any big mistakes the AI safety community has made in the past or are currently making?"
"Yeah, probably most things people are doing are mistakes. This is just some random group of people. Why would they be making good decisions on priors? When I look at most things people are doing, I think they seem not necessarily massively mistaken, but they seem somewhat confused or seem worse to me by like 3 times than if they understood the situation better." - Ryan Greenblatt
"If we look at the track record of the AI safety community, it quite possibly has been harmful for the world." - Adam Gleave
"Longtermism was developed basically so that AI safety could be the most important cause by the utilitarian EA calculus. That's my take." - Holly Elmore
Participants pointed to a range of mistakes they thought the AI safety movement had made. Key themes included an overreliance on theoretical argumentation, being too insular, putting people off by pushing weird or extreme views, supporting the leading AGI companies, insufficient independent thought, advocating for an unhelpful pause to AI development, and ignoring policy as a potential route to safety.
How to read this post
This is not a scientific analysis of a systematic survey of a representative sample of individuals, but my qualitative interpretation of responses from a loose collection of semi-structured interviews. Take everything here with the appropriate seasoning.
Results are often reported in the form "N respondents held view X". This does not imply that "17-N respondents disagree with view X", since not all topics, themes and potential views were addressed in every interview. What "N respondents held view X" tells us is that at least N respondents hold X, and consider the theme of X important enough to bring up.
The following is a summary of the main themes that came up in my interviews. Many of the themes overlap with one another, and the way I've clustered the criticisms is likely not the only reasonable categorization.
Too many galaxy-brained arguments & not enough empiricism
"I don't find the long, abstract style of investigation particularly compelling." - Adam Gleave
9 respondents were concerned about an overreliance or overemphasis on certain kinds of theoretical arguments underpinning AI risk: namely Yudkowsky's arguments in the sequences and Bostrom's arguments in Superintelligence.
"All these really abstract arguments that are very detailed, very long and not based on any empirical experience. [...]
Lots of trust in loose analogies, thinking that loose analogies let you reason about a topic you don't have any real expertise in. Underestimating the conjunctive burden of how long and abstract these arguments are. Not looking for ways to actually test these theories. [...]
You can see Nick Bostrom in Superintelligence stating that we shouldn't use RL to align an AGI because it trains the AI to maximize reward, which will lead to wireheading. The idea that this is an inherent property of RL is entirely mistaken. It may be an empirical fact that certain minds you train with RL tend to make decisions on the basis of some tight correlate of their reinforcement signal, but this is not some fundamental property of RL."
Alex Turner
Jamie Bernardi argued that the original view of what AGI will look like, nam...