LW - Live Theory Part 0: Taking Intelligence Seriously by Sahil

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Live Theory Part 0: Taking Intelligence Seriously, published by Sahil on June 27, 2024 on LessWrong.
Acknowledgements
The vision here was midwifed originally in the wild and gentle radiance that is Abram's company (though essentially none of the content is explicitly his).
The PIBBSS-spirit has been infused in this work from before it began (may it infuse us all), as have meetings with the Agent Foundations team at MIRI over the past ~2 years.
More recently, everyone who has been loving the High Actuation project into form (very often spontaneously and without being encumbered by self-consciousness of this fact):[1] individuals include Steve Petersen, Mateusz Baginski, Aditya Prasad, Harmony, TJ, Chris Lakin; the AISC 2024 team, Murray Buchanan, Matt Farr, Arpan Agrawal, Adam, Ryan, Quinn; various people from Topos, ALIFE, MAPLE, EA Bangalore. Published while at CEEALAR.
Disclaimers
Very occasionally there are small remarks/questions from a remarkable human named Steve, since this and the next two posts are an edited transcript of me giving him a talk. I left them in to retain the conversational tone. Steve has also consistently been a fantastic ground for this channeling.
I use the term "artefact" a fair amount in this sequence. Unfortunately for you and me, Anthropic also recently started using "artifact" in a different way. I'm using "artefact" in the common sense of the word. The British spelling should help remind of the distinction.
Taking Intelligence Seriously
Sahil:
I gave a talk recently, at an EA event just two days ago, where I made some quick slides (on the day of the talk, so not nearly as tidy as I'd like) and attempted to walk through this so-called "live theory". (Alternative terms include "adaptive theory" or "fluid theory"; where the theories themselves are imbued with some intelligence.)
Maybe I can give you that talk. I'm not sure how much of what I was saying there will be present now, but I can try. What do you think? I think it'll take about 15 minutes. Yeah?
Steve:
Cool.
Sahil:
Okay, let me give you a version of this talk that's very abbreviated.
So, the title I'm sure already makes sense to you, Steve. I don't know if this is something that you know, but I prefer the word "adaptivity" over intelligence. I'm fine with using "intelligence" for this talk, but really, when I'm thinking of AI and LLMs and "live" (as you'll see later), I'm thinking, in part, of adaptive. And I think that connotes much more of the relevant phenomena, and much less controversially.
It's also less distractingly "foundational", in the sense of endless questions on "what intelligence means".
Failing to Take Intelligence Seriously
Right. So, I want to say there are two ways to fail to take intelligence, or adaptivity, seriously.
One is, you know, the classic case, of people ignoring existential risk from artificial intelligence. The old "well, it's just a computer, just software. What's the big deal? We can turn it off." We all know the story there. In many ways, this particular failure-of-imagination is much less pronounced today.
But, I say, a dual failure-of-imagination is true today even among the "cognoscenti", where we ignore intelligence by ignoring opportunities from moderately capable mindlike entities at scale. I'll go over this sentence slower in the next slide.
For now: there are two ways to not meet reality.
On the left of the slide is "nothing will change". The same "classic" case of "yeah, what's the big deal? It's just software."
On the right, it's the total singularity, of extreme unknowable super-intelligence. In fact, the phrase "technological singularity", IIRC, was coined by Vernor Vinge to mark the point that we can't predict beyond. So, it's also a way to be mind-killed. Even with whatever in-the-limit proxies we have for this, we make various sim...