“Gradual Disempowerment, Shell Games and Flinches” by Jan_Kulveit

Over the past year and half, I've had numerous conversations about the risks we describe in Gradual Disempowerment. (The shortest useful summary of the core argument is: To the extent human civilization is human-aligned, most of the reason for the alignment is that humans are extremely useful to various social systems like the economy, and states, or as substrate of cultural evolution. When human cognition ceases to be useful, we should expect these systems to become less aligned, leading to human disempowerment.) This post is not about repeating that argument - it might be quite helpful to read the paper first, it has more nuance and more than just the central claim - but mostly me ranting sharing some parts of the experience of working on this and discussing this.

What fascinates me isn't just the substance of these conversations, but relatively consistent patterns in how people avoid engaging [...]

---

Outline:

(02:07) Shell Games

(03:52) The Flinch

(05:01) Delegating to Future AI

(07:05) Local Incentives

(10:08) Conclusion

---

First published:
February 2nd, 2025

Source:
https://www.lesswrong.com/posts/a6FKqvdf6XjFpvKEb/gradual-disempowerment-shell-games-and-flinches

---

Narrated by TYPE III AUDIO.