Definitely. Keep up the great work! Live long and prosper. 🖖

Amazing. Blackpool North. That sounds remarkably accessible from Oxford.

Definitely. As you know, I’m new to the technical part of the field, though I’ve been in the policy part for a long time. I will be spending much more time visiting technical communities. I literally started with the FHI folks in Oxford, so if there are other high-value ones, ...

Yes. Let’s make it happen.

Amazing. Well, I think we’re in the lucky timeline — we only took an hour but managed to align on everything we have talked about. So obviously, alignment is possible between researchers.

And also give us shared vocabulary. A lot of this is reminiscent of theoretical physics, where a good experiment can unify many different theoretical branches because they suddenly realize they’re talking about the same thing.

I think the hope of the Metta AI simulator is that at least the parts around group dynamics and takeover scenes—they even have a scenario called “Clips versus Cogs,” where the Clips try to turn everything into paperclips and Cogs try to defend—can at least put some of these technical ...

Maybe they’re both right and the world ends either way, but yes.

Indeed, the same technology that translates “climate justice” to “creation care” and back can be used to translate among the different alignment vocabularies.

I agree. This could also be a good bootstrap phase toward CEV, assuming CEV is achievable.

Like participatory design over lightcones.

If, at that point, we figure out that it is technically impossible to climb another ladder, then a strong, horizontally aligned group intelligence would stop there instead of committing suicide. Which is one of the good endings.

I think what CBV means by “blending” is not averaging. Each human has an incommensurable, irreducible set of preferences. Instead of extrapolating, which will only diverge, we need to figure out a way we can “live and let live,” and make this agreement coherent such that everybody in a hyperlocal ...

Well, that is not different from the way we would explain today to humans from two hundred years ago?

Are you familiar with the Ben Goertzel idea from 2012? The main idea is that instead of extrapolating human volition, which allows a razor-thin margin of error and is very difficult anyway, we focus on the capacity of humans to blend their volitions over time.

This would manifest not as slaves bound to single human principals, which doesn’t work, but rather as a team coach or facilitator that works toward a fiduciary duty to the <em> relationship </em> between human actors. The core insight is that loyalty is not to single human preferences or any ...

A core insight is that We, the People, are <em> already </em> the superintelligence we’re looking for. I’m picturing the latent human communities as superintelligence. What is needed for superhuman-level strengths is not something outside our moral scope, but something we can always reason about in a hyperlocal way.

Okay. First, I remember the phrase you used: “applause lights.” I will try to keep applause lights to zero in my answer.

I’m just happy to calibrate our wordings. A lot of written words can be interpreted in various ways. Almost like a late Wittgensteinian “fly out of the bottle.”

The intuition here is that we are composed of “dividuals”—shards or whatever—that are specimens of multiple different hyperlocalities. If we achieve some sort of internal coherence through negotiation, then it emerges in each bottom-up level. Hyperlocal morality gets resolved at a slightly higher level, but never at the very top ...

Yes, I did.

Then it sounds to me the obvious way out is to simply show very clear human preferences along the way, such that one cannot avoid seeing the divergence. You can have hyperlocal and regional dashboards; you can measure, for example, polarization per minute. Instead of each AI-human pair “reading the ...

This assumes that if the system today already rewards outcomes that are bad for humans, AI systems may be more effectively following these incentives locally. Humans, wanting short-term profit, delegate important judgments to the AI. And when millions of people do so to a small degree, it all adds up ...

So what you’re saying is that the gradual disempowerment can be so gradual that each move doesn’t look like it’s violating boundedness, but over a long time, it adds up to disempowerment?

I’m trying to play this out. Like, what do you think the leaders of major labs are really doing?

I’m reminded of Shahar Avin’s BAK game, in which you can’t really tell from the first few plays whether somebody is trying to defect or not, because their plays can be interpreted both ways.

In a d/acc world where we figure out defense dominance, it becomes almost trivially solvable. In the reality we are in, we need a two-layer protocol: start the norm around the defense-dominant pods, and then transfer the norm so the offense-dominant parts are still bound by the larger players. You ...

Okay, so that moves to the second part: whether it’s impossible to stop rogue actors. I think we differ on how many of the main players are sufficient to play by the rules in order to police against pure unbounded players.

We are in a similar situation. Most AI systems today that deliver economic value require some sort of steerability and boundedness. Most manufacturers would at least agree to pay lip service to managing externalities, because large negative externalities destroy the general direction of the capitalist instrument.

Let’s tease those apart. For Freon, people didn’t have perfect, drop-in replacements just waiting on the shelf. The Montreal Protocol was a technology-forcing regulation: It created a firm deadline that compelled companies to invest heavily to commercialize and scale up known but underdeveloped alternatives. The global commitment to the phase-out ...

But if that is true, why did people stop manufacturing Freon?

The “goes ahead” part I’d worry less about, if we make it common knowledge that steerable, bounded systems can offer better economic returns. This way, building unbounded systems becomes irresponsible—like manufacturing Freon, which destroyed the ozone—and would not get the investment they are getting now.

Then, we use those more coordinated communities to create what I call “Kamis” (local stewards), which are strategy-proof instantiations that only care about their particular community. This is a ladder where you build a firm, concrete ground before moving to the next stage.

My main point was that we do have the tools to align current-day systems, and we should practice our civic muscles so we have at least good enough control over these algorithms. Only then should we move toward improving our coordination mechanisms, using the already aligned systems, but only in ...

It sounds like we agree on this part as well.

Which is why I think we agree that, if we cannot even align current-day recommendation systems, we shouldn’t try to move to the next ladder.

Which is arguably already happening. This is what I mean by parasitic recommendation systems.

Okay. That sounds quite similar to what I had in mind for strategy-proofness. Basically, the sharp-left-turn generating dynamics need to be not just improbable, but information-theoretically impossible. The rules we make toward each step of bootstrapping need to be immune to manipulation, even by coalitions of otherwise aligned players.

When you say “robust,” do you mean it in the game-theoretic sense as being “strategy-proof”—that no matter what you do, you cannot move outside of this equilibrium?

That’s the d/acc idea: keep the defense-dominant communities on the technically advanced track until they can complete the Culture-esque bootstrap. It seems we agree generally on the winning condition, but the backcasting has important holes.

Okay. So that sounds to me like we need to model communities as first-class. We start with the existing communities of technically capable humans and their narrow AI systems—we are essentially partial cyborgs already—and bootstrap them.

That I agree with. Essentially, even if the community that cares about civic relational health figures out a stable governance system, if an “ironclad” comes in and there’s no way to punish it, then whomever retained resources and did not use them to tend their gardens ends up winning industrialization ...

So it’s like, to be a good citizen, you have to have a garden. But if I want to game this, I just do a minimally viable garden, maybe with plastic. Essentially what Eliezer Yudkowsky argued—just keep humanoids in a cage and hope other Minds cannot tell the difference?