My point is closer to this: if you have a system where the top-level agents do <em> not </em> have stabilized value systems that take care of humans, then competitive dynamics take over. If the Minds were all out for themselves and playing power-seeking games, the Minds that spend fewer ...

Not quite. I model the Culture as a kind of singleton—not a single overseeing mind, but in the sense that each individual Mind has something like a value handshake or merged values. If one Mind started doing something repulsive, the others would unite and stop it. The entire collective has ...

I don’t think you get something that a person today would accept. They would say, “Wow, that’s horrifying.” Even if the humans aren’t literally all dead, they end up as irrelevant as the larger agents find convenient—which is probably almost entirely irrelevant.

My best guess is that if you have something like this, you end up with the human being basically irrelevant. In competitive dynamics, the larger agents which discard or reduce human input to their decision procedures end up winning, because the humans are just an unnecessary appendage in the wider ...

If you have a system with a mixture of more and less capable agents, you will tend towards the more capable agents strongly dominating. If you have an ecosystem with human-level agents and greatly superhuman agents, without strong principles protecting the humans, I think the humans end up getting crushed ...

One is a model of agency as taking the logical shape of the future and branching towards preferred states. A stronger agent will see further into the future and be able to steer more effectively. But this is recursive, in that it can steer towards states where it’s better able ...

Yep. I can point to a few dynamics.

Cool. I’ve got a very rough sketch of Emmett’s work, but not a detailed understanding. At a high level, I don’t see how this could work, but I don’t know exactly what he’s trying, so perhaps there’s a trick I didn’t think of that makes this viable.

So first, before I list the things I’d be excited to talk about, I just want to say I’m very impressed by the things you’ve done. I rarely see people who deeply get mechanism design and also how to shift the levers of the world—and understand that yelling at people ...