LW - The problem with infohazards as a concept [Linkpost] by Noosphere89

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The problem with infohazards as a concept [Linkpost], published by Noosphere89 on December 22, 2023 on LessWrong.
This is going to be a linkpost from Beren on some severe problems that come with embracing infohazards as a useful concept.
The main problem I see that are relevant to infohazards are that it encourages a "Great Man Theory" of progress in science, which is basically false, and this still holds despite vast disparities in ability, since no one person or small group is able to single handedly solve scientific fields/problems by themselves, and the culture of AI safety already has a bit of a problem with using the "Great Man Theory" too liberally.
There are other severe problems that come with infohazards that cripple the AI safety community, but I think the encouragement of Great Man Theories of scientific progress is the most noteworthy problem to me, but that doesn't mean it has the biggest impact on AI safety, compared to the other problems.
Part of Beren's post is quoted below:
Infohazards assume an incorrect model of scientific progress
One issue I have with the culture of AI safety and alignment in general is that it often presupposes too much of a "great man" theory of progress 1 - the idea that there will be a single 'genius' who solves 'The Problem' of alignment and that everything else has a relatively small impact. This is not how scientific fields develop in real life. While there are certainly very large individual differences in performance, and a log-normal distribution of impact, with outliers having vastly more impact than the median, nevertheless in almost all scientific fields progress is highly distributed - single individuals very rarely completely solve entire fields themselves.
Solving alignment seems unlikely to be different a-priori, and appears to require a deep and broad understanding of how deep learning and neural networks function and generalize, as well as significant progress in understanding their internal representations, and learned goals. In addition, there must likely be large code infrastructures built up around monitoring and testing of powerful AI systems and an sensible system of multilateral AI regulation between countries.
This is not the kind of thing that can be invented by a lone genius from scratch in a cave. This is a problem that requires a large number of very smart people building on each other's ideas and outputs over a long period of time, like any normal science or technological endeavor. This is why having widespread adoption of the ideas and problems of alignment, as well as dissemination of technical work is crucial.
This is also why some of the ideas proposed to fix some of the issues caused by infohazard norms fall flat. For instance, to get feedback, it is often proposed to have a group of trusted insiders who have access to all the infohazardous information and can build on it themselves.
However, not only is such a group likely to just get overloaded with adjudicating infohazard requests, but we should naturally not expect the vast majority of insights to come from a small recognizable group of people at the beginning of the field. The existing set of 'trusted alignment people' is strongly unlikely to generate all, or even a majority, of the insights required to successfully align superhuman AI systems in the real world.
Even Einstein - the archetypal lone genius - who was at the time a random patent clerk in Switzerland far from the center of the action - would not have been able to make any discoveries if all theoretical physics research of the time was held to be 'infohazardous' and only circulated privately among the physics professors of a few elite universities at the time. Indeed, it is highly unlikely that in such a scenario much theoretical physics would have been done at all.
Similarly,...