Yep. A lot of Lighthaven events are also great. Incredible speaking with you, and yeah, glad to have you on the team working to save the world.

Also, the LessWrong community weekend happens around September most years and has a high concentration of interesting people.

Awesome. Happy to continue to collaborate. Let me know if there’s anything you’d find useful. I have a pretty strong network having worked in this field for a decade.

It is possible if both people are running a process that’s able to receive and translate. Having that be scalable to a bunch of other people seems very good if possible.

That would be the dream.

You can definitely run small experiments, but there are going to be important distributional shifts between toy systems and extremely capable situationally aware systems that know they can defeat you. You want a robust theory that you have reason to expect to scale to the real world, rather than relying ...

Yeah. And I imagine there’s some extra mechanism design needed to make technical coherence in a domain where you can’t get a clear reward signal, because you can’t check whether your plan will work without testing the thing, and if you test it, whoops, if you got it wrong.

Yep. And there are probably extra pieces of mechanism design needed where some stuff gets deep into very technical weeds. We need a way to cause people to engage across ontological gaps and strong differences—where one person believes if we do ’A’, the world will end, and another believes if ...

Getting to the point where the people trying to solve the large-scale problem (and not just local puzzles without a broad strategic view) can reliably speak the same language is something we are incredibly far away from. This results in wasted effort, frustration, and infighting. We as a community are ...

Absolutely agree. The second point is that the alignment field itself is astonishingly bad at sense-making about what the technical problem is or how to address it. If you take three alignment researchers and put them in a room, you will have more than three opinions, and not just different ...

Manually do a little bit of CEV ourselves. I funded a friend to make a series of interviews called “Utopiography,” interviewing people about what they would like the future to be like. But something much more scalable seems way better.

The first is humanity’s sense-making about where we want to go. We’re going to have technology that could do truly profound things—solve medicine, aging, climate change, etc. We as a species haven’t figured out what we want to do with that. The kind of mechanism design you’ve developed might be ...

Yep, this seems pretty healthy. There are a couple of things where what the ecosystem needs feels close to the skill set you’ve developed.

Yes. We realize we are not ready to take the next step, or it is physically impossible, and therefore, we don’t. This seems pretty healthy, though also very ambitious. Getting to a system where we’re so coordinated that we could both notice we can’t safely take another step and then ...

I agree this vision sounds really good. I still have the intuition that if I’m understanding this right, you don’t end up anything like that. Maybe the thing I’m missing is that what you’re pointing to is not the final thing that defends humane values, but instead, this is the ...

No, I think it’s much worse. A person from two hundred years ago would be confused. But if you just blend the volition in a way that doesn’t preserve and protect individual agency and human values robustly, I think it falls to natural gradients into something much more alien.

If we imagine looking at a world two hundred years after this has been done and describing it to someone alive today, do you think that person is profoundly horrified?

Mhmm. Why isn’t this clicking into place?

Okay. My core question is: In an organic system with some human-level and some strongly superhuman-level systems, what stops the humans from losing influence over time to the point that they lose the ability to sustain themselves? By default, I think this doesn’t work. What’s the core insight that makes ...

Cool. Thanks. Let’s defer zooming in on that until I’ve read more. Let’s also zoom out. Which bits of conversation feel like the most high value?

It’s not implausible that works. I don’t feel confident the system doesn’t end up decaying into weird states because I don’t understand how the scaling up defends against some of the failings. I’m interested in references on the technical philosophy going on here.

Okay, this is plausible to me. My main concern is that if you solve all the hyperlocal moral philosophies, the system as a whole does not converge towards something that most participants at the start would have thought is good. You might end up with parts of the hyperlocal system ...

I think the version of this that’s truly robust ends up looking less like a bunch of handcrafted metrics and more like, “We solved moral philosophy, and from this, it is obvious what all the metrics should be.” This is a fairly hard problem. Otherwise, if you’re just having semi-arbitrary ...

Yes, this captures a bunch of it.

Yes. Although I think that even though it’s gradual in some sense, it can be relatively quick. It can be lots of steps in quick succession and accelerating—a race to the bottom that might be as short as low single-digit years. Even if no actor ends up skipping the “gradual” ...

It’s not really a plot to reach an end state that is generally considered bad. I think they generally are able to think it’s for the greater good, or the least bad option.

That sounds like a similar dynamic.

Increasingly, capital resources and influence are allocated towards subsystems that are doing more competition than their competitors. Those subsystems will end up dominating the narrative. I see bits of this happening with some of Sam Altman’s plays. It feels like the kind of preemptive plays of someone who wants to ...

Yes, but additionally I can make a stronger case. It’s not just defect versus cooperate, but a spectrum. The further you move along the spectrum towards unbounded optimizing strong systems, the more likely you are to win dominance. Detecting how far someone is along that spectrum is much more expensive ...

On some level, people prefer to maintain control, but also a lot of people would say, “Please make me a billion-dollar company. Go do it yourself.” The natural gradient falls towards humans offloading more and more of their cognition and decisions into increasingly broadly agentic systems which are doing the ...

I agree many would pay lip service, but I think this is insufficient to prevent <em> anyone </em> from building an unbounded system when that system will effectively outcompete bounded systems.

One, they had reasonably good replacements; it didn’t cost them very much. Two, there was enough pressure. It is possible to restrict things that are locally mildly economically productive, but we don’t have examples of technologies on anything like the scale of economic and strategic importance as this one being ...

Unfortunately, I think that’s false. I think unbounded systems produce much larger economic returns. The less bounded a corporation’s AI systems are, the more profit they will get. At some point, the system becomes so capable it kills all the humans, but in some sense, it still has more profit. ...

I like all of these steps until the last. I’m worried that the bounded Kami thing runs into unsolved, maybe unsolvable, technical problems at high power levels. If you have a reasonably powerful cognitive system with some level of self-authorship, I think it converges towards greater clarity of thought and ...

Yes. We are unprepared for this technical problem. We’ve made real progress, but only a small fraction of the progress we would need.

Yes. It just gets a lot worse when the system is agentically superintelligent.

Yes. This is one interpretation. Another angle: At a sufficiently powerful cognitive level, humans trying to steer where to go becomes unviable. It doesn’t represent our values as we intend, because if the AI has any preferences of its own mixed in, it can give us inputs that cause us ...

I can clarify. Robust here means something like: if it interacts with an environment that has other AIs trying to manipulate it, it stays pointed at good things (unless those AIs are dramatically more capable). You can’t use the system to build an unaligned system. The system won’t fall out ...

Without a robust and general theory of how to go up that curve indefinitely, you probably miss a step. It might be subtle; maybe one of your systems is now power-seeking in the background. A few steps out, you end up with dramatically more capable systems that do not deeply ...

Yep. We both want a stable system at the top which stops very bad things from happening, while leaving room for flourishing agency within it. But how to get there is the question.

Yeah, this is the core dynamic.

This is avoidable, but only if the top level crystallizes around codified defense, like, “We together as a group have decided we wish to protect these values.” Otherwise, the systems that don’t share those values but still want to win, win step by step over a potentially large time period.

It’s not that they wouldn’t be able to tell the difference. It’s that there’s not a unified top-level system for punishing defectors. The claim is that without a robust system at the top level, the weeds grow step by step.