Yes, I did.

Then it sounds to me the obvious way out is to simply show very clear human preferences along the way, such that one cannot avoid seeing the divergence. You can have hyperlocal and regional dashboards; you can measure, for example, polarization per minute. Instead of each AI-human pair “reading the ...

This assumes that if the system today already rewards outcomes that are bad for humans, AI systems may be more effectively following these incentives locally. Humans, wanting short-term profit, delegate important judgments to the AI. And when millions of people do so to a small degree, it all adds up ...

So what you’re saying is that the gradual disempowerment can be so gradual that each move doesn’t look like it’s violating boundedness, but over a long time, it adds up to disempowerment?

I’m trying to play this out. Like, what do you think the leaders of major labs are really doing?

I’m reminded of Shahar Avin’s BAK game, in which you can’t really tell from the first few plays whether somebody is trying to defect or not, because their plays can be interpreted both ways.

In a d/acc world where we figure out defense dominance, it becomes almost trivially solvable. In the reality we are in, we need a two-layer protocol: start the norm around the defense-dominant pods, and then transfer the norm so the offense-dominant parts are still bound by the larger players. You ...

Okay, so that moves to the second part: whether it’s impossible to stop rogue actors. I think we differ on how many of the main players are sufficient to play by the rules in order to police against pure unbounded players.

We are in a similar situation. Most AI systems today that deliver economic value require some sort of steerability and boundedness. Most manufacturers would at least agree to pay lip service to managing externalities, because large negative externalities destroy the general direction of the capitalist instrument.

Let’s tease those apart. For Freon, people didn’t have perfect, drop-in replacements just waiting on the shelf. The Montreal Protocol was a technology-forcing regulation: It created a firm deadline that compelled companies to invest heavily to commercialize and scale up known but underdeveloped alternatives. The global commitment to the phase-out ...

But if that is true, why did people stop manufacturing Freon?

The “goes ahead” part I’d worry less about, if we make it common knowledge that steerable, bounded systems can offer better economic returns. This way, building unbounded systems becomes irresponsible—like manufacturing Freon, which destroyed the ozone—and would not get the investment they are getting now.

Then, we use those more coordinated communities to create what I call “Kamis” (local stewards), which are strategy-proof instantiations that only care about their particular community. This is a ladder where you build a firm, concrete ground before moving to the next stage.

My main point was that we do have the tools to align current-day systems, and we should practice our civic muscles so we have at least good enough control over these algorithms. Only then should we move toward improving our coordination mechanisms, using the already aligned systems, but only in ...

It sounds like we agree on this part as well.

Which is why I think we agree that, if we cannot even align current-day recommendation systems, we shouldn’t try to move to the next ladder.

Which is arguably already happening. This is what I mean by parasitic recommendation systems.

Okay. That sounds quite similar to what I had in mind for strategy-proofness. Basically, the sharp-left-turn generating dynamics need to be not just improbable, but information-theoretically impossible. The rules we make toward each step of bootstrapping need to be immune to manipulation, even by coalitions of otherwise aligned players.

When you say “robust,” do you mean it in the game-theoretic sense as being “strategy-proof”—that no matter what you do, you cannot move outside of this equilibrium?

That’s the d/acc idea: keep the defense-dominant communities on the technically advanced track until they can complete the Culture-esque bootstrap. It seems we agree generally on the winning condition, but the backcasting has important holes.

Okay. So that sounds to me like we need to model communities as first-class. We start with the existing communities of technically capable humans and their narrow AI systems—we are essentially partial cyborgs already—and bootstrap them.

That I agree with. Essentially, even if the community that cares about civic relational health figures out a stable governance system, if an “ironclad” comes in and there’s no way to punish it, then whomever retained resources and did not use them to tend their gardens ends up winning industrialization ...

So it’s like, to be a good citizen, you have to have a garden. But if I want to game this, I just do a minimally viable garden, maybe with plastic. Essentially what Eliezer Yudkowsky argued—just keep humanoids in a cage and hope other Minds cannot tell the difference?

Let me play this back using the example of an Iain M. Banks Culture ship run by a Mind. You’re essentially saying that, even though the Mind in the novel is portrayed as seeing humans as curious, fun, or symbiotic like pets, so they may genuinely prefer a continued relationship ...

I’m aware of that line of argument, and I think it’s largely orthogonal. When we are talking about the symbiogenetic part, this assumes the endosymbiosis idea, where the larger agent subsumes the smaller agent into part of itself. From a mitochondrion’s point of view, they still have agency in some ...

Yes, indeed. And I also understand your concerns about how going into too much detail about certain future simulations may actually inadvertently make them real—a kind of hyperstition. So I am entirely fine with this transcript being a co-curated work. We will co-edit the transcript, and only when we’re both ...

To the honorable members of the Diet, Minister Taira, MP Anno, members of Team Mirai and all our friends in the media, good morning. Thank you for this opportunity.

As you know, we all live in an era of high PPM. I do not mean parts per million of carbon, ...

In late September, a U.S. presidential executive order set the final terms for TikTok's American operations: its parent company, ByteDance, must divest, and control must pass to a U.S.-led investment group spearheaded by Oracle and Dell. This "Divest-or-Ban" deal has finally been settled.

Foreign-led services are ubiquitous, so why is ...

But no democracy is an island, not even Taiwan. So, the other laureates show across the world in various different configurations, in the front line in the struggle against autocracy that again and again, horizontal peer-to-peer, this horizontal mode of trust can actually get depolarization done, can deliver results. And ...

Well, first of all, I’m really, really honored to be recognized this way. I’m truly humbled by the 203 laureates. And I think we unite in a mission to make sure that the depolarization methods are widely known and shared. I think this is very important because Taiwan researched these ...

And in a citizen assembly, especially using bridging algorithms, this gives virality to the middle ground, to the uncommon ground that is collectively discovered. And so once again, I think this depolarization is really key. After hundreds of such exercises, the people of Taiwan understand that democracy is not just ...

This is the beauty of bridge-making systems. For example, community notes, which was inspired by the Polis algorithm that we tried in Taiwan and still use in Taiwan. It’s not just the left wing in this bubble or the right wing in this bubble, it is calculating the up-wing, the ...

Well, first of all, again, if we take the protocol-based innovation lens, then it is positive-sum, right? If TikTok offers its videos in an open protocol and makes sure that anybody with any better algorithm can rank those videos according to people’s preferences instead of a centralized preference, then yes, ...

Well, first of all, I would say that it is a matter of network flow analysis. This is a matter that independent audits can easily prove. And, so I think in internet governance, what is important is that independent third parties can also monitor and see for themselves whether the ...

And the idea again is that in Taiwan we have for a very long time now since 2019, I believe, that says that anything that is de facto controlled by Beijing should actually not get adopted without question in our public sector. Whether to integrate it more or less, is ...

And, because of that, we share this idea. And during Trump 1.0, the idea gained this branding called a “Clean Network,” I believe. And then this idea got adopted by many democratic partners. Now in Trump 2.0, and I would say also that Biden continued this idea of a trustworthy ...

So, one of the ideas that Taiwan had when we had half a million people on the street and many more online in 2014 is to build this bridging consensus when it comes to whether to allow the Beijing brands like Huawei and ZTE into our then new 4G network. ...

Yes, definitely. So, the tools that I just described, such as Polis, the Google Jigsaw Sensemaker, the Frankly online deliberative polling software, many others. All of them are not just available, but rather free, as in free software, so that you can customize it however you want. And if you’re ...

And so I think the idea here is again to radically trust the people and show the big tech companies that this is the measure that does not sacrifice freedom of expression. But the freedom of speech does not mean that we need to get foreign robots freedom of reach ...

In Taiwan last March, to solve the problem of advertisers pushing deep fake scams. We sent 200,000 text messages to random numbers around Taiwan asking how we should make sure that the information integrity is upheld without harming freedom of expression because after all Taiwan is the most free in ...

Well, first of all, we need to distinguish speech versus reach. Of course, speech needs to be protected. On the other hand, currently in social media, we’re seeing a situation where if I don’t subscribe to someone, but if the algorithm think that this awful speech will cause me to ...

So, to your question, I believe it is very important that Europe also look at social portability like in Utah and also we can figure out a way to do portability across AI services as well. Well, I mean the idea of telecom portability, if you switch from telecom A ...

Here in the U.S., the state of Utah also passed the Digital Choices Act that goes a step further. Starting next July, if you are a Utah citizen, you can move from social media A to social media B, while keeping all your community. The new likes, reactions, replies will ...

I think what Europe has, which is this idea of portability, is very important. As you know, currently with the Digital Markets Act, you can have instant message systems coming from different companies talking to one another, so they must compete on better service instead of on the capture of ...

And what we found out is that the tools such as Polis, broad listening tools, Sensemaker, and so on, can lead to a much more accurate poll, which is not predetermined, but rather a generative poll, where the options are generated by the citizens, or what I call pro-social media. ...