Okay, so you… and you’ve ridden with me, and I call that riding on the doom train. You’ve accepted that super intelligence is real, so yeah, that’s right. And then we start diverging as soon as we talk about alignment, because you’re like: “Yeah, you know, we’ll have time. The ...

Yes, a foomer. Right, a foomer bloomer. Not a doomer. Okay, that’s a good way to describe it. Yeah.

You’re like: “Yep, if you just pluck the ideal super intelligent algorithm out of algorithm space, it can recursively self-improve, and if it’s not aligned to humanity, it can definitely kill everybody.”

So my attempt at summarizing your perspective is just like, well, first of all, it’s very interesting that you are kind of on the foom train. You’re like, “Oh, yeah, foom could very much be real. Intelligence is quite a high spectrum,” which I appreciate that you have basically ridden ...

All right. Sounds good. So this was a great conversation. At the end, I basically just summarize the disagreement, summarize what the guest communicated was their perspective, and why it’s not just equivalent to my own perspective.

Whereas anybody with a P(Doom) of, I don’t know, thirty percent plus, you’d be like: “Hell, yeah, we need an off button because there’s a thirty percent chance that this thing is going to doom us so soon. Of course, we need the ability to turn it off.” So that’s ...

Okay, I mean, what I’m getting is just that your P(Doom), I know you won’t say it or think it’s meaningful, but I model you as effectively having a P(Doom) that’s, let’s say, ten percent, in the ballpark of ten percent. And because of that, you’re like, “Eh, so if ...

I feel like most would be like, “Okay, all of us get cancer… yeah, I would measure that in decades.” Whereas in the case of AI, the experts are saying… a lot of them are saying, I don’t know, five years? Less than five years, maybe. Could even be next ...

So in that particular analogy, though, if you just asked a bunch of experts, even if they’re all hand-waving, be like: “Look, we don’t know when the ozone’s gonna get depleted.” Be like, “Okay, just give me a vague distribution. If we go on this pace, how long before we ...

So isn’t it a priority? Do you think it’s a policy priority? Out of the many… you’re talking about red lines, should we also potentially architect an off button that a few countries, the most powerful countries in the AI space, can vote to be like, “All right, shut it ...

Yeah, so if you’re all for drawing these red lines, let me just ask you very specifically, because my favorite policy proposal, the thing that feels like the biggest win to me, is this idea of preparing to potentially pause. “Hey, we’re all looking at this event, the singularity. We ...

You know, the only other thing that often comes up about China is this idea of the international treaty. It’s probably not something that you think much about because you seem pretty optimistic about the foom going well. Do you have any opinion on the pause AI movement and then ...

Right. Okay. You know, if you read some of these latest things, like situational awareness, and there was, I forgot what it’s called, there was another paper saying: “Yep, countries are really going to be going after each other’s AI powers, and this is such a critical node.” You know, ...

So your various efforts have strengthened Taiwan in a way that reduces its isolation from the larger world. So you strategically made those kind of efforts, which I think is great. You’ve reduced its dependence on China in some ways. I’ll open the floor to you. What is your view ...

And if they were to do that, maybe China can prevent the US from having more access to TSMC chips than they do. This will intensify as AI companies become worth exponentially more. AI companies are, in my opinion, going to be worth tens of trillions pretty soon. So if ...

But as we know, the most valuable physical manufacturer of chips, Taiwan Semiconductor Manufacturing Corporation, with a $1.6 trillion dollar market cap, TSMC, for the viewers, it takes Nvidia’s blueprints, and it physically constructs the chips using advanced lithography machines. TSMC owns the fabrication plants, and one of the hottest ...

All right, but yeah, heading into the wrap-up here, let me try briefly asking you about the third rail here, China geopolitics, and obviously feel free to say what you can, don’t say what you can’t. It’s a bit tangential to the AI debate, because I think all of humanity ...

Okay. Yeah, fair enough. My objection is just that I think it’s going to run out of control before we can calibrate that dynamic. We just don’t get enough iterations. We don’t get enough trial and error. We just lose control, and it’s way too late. That’s my mainline scenario.

Right. Okay, so you’re just saying, yeah, Hinton’s kind of making sense when he talks about AI being our mother, but it’s a little bit more analogous to say that AI is gonna be our dog mom or our tomato mom.

But you actually, I think, disagree with that. You don’t think it’s plausible to make AI maternal, and instead, you prefer the analogy of making AI like our gardener? Okay, explain that.

Hinton, over the last few months, used to feel really pessimistic, but now he’s saying, “Hey, I realize that we just need to program some kind of maternal instinct into the AI, the same way that human mothers are aligned with their children, because they have this maternal instinct. They ...

Okay, I mean, if we have all this time… if we have enough time, then I don’t have a strong opinion to agree or disagree with you. If we get undos, if we don’t all die, should we strive to make alignment top-down, or should we strive to make it ...

When I think about ideal alignment, I do think about, like, coherent extrapolated volition. It is kind of top-down, where once you’ve sufficiently understood the true utility function of humanity, there’s nothing fundamentally wrong with just top-down installing that, is there?

And that gets to this whole thing that I agree with… yeah, it’s complicated. Alignment itself is a complicated problem. But I wouldn’t describe that as a failure of top-down alignment. I would just describe that as a failure to specify a sufficiently subtle utility function, but it could still ...

Yeah, I mean, that is true. You can imagine, you know, if it’s like you might wanna gamble 0.01 percent increase in human extinction in exchange for rapid increases in science or whatever. So I totally agree when the percent is low enough, sure.

Okay. So going back to this thesis that you’re saying, that AI alignment cannot be top-down, what if we just throw in some top-down instructions to AI, for example, like, try to not make humanity go extinct? Isn’t that a pretty good top-down directive?

Right. So if I’m understanding correctly, you would have preferred if OpenAI kind of gives this open-ended, open source… and you’re even saying you like their open source model. So you’re basically saying they should add more user-tweakable parameters so that everybody can align it to their taste more powerfully?

Yeah. Could we do a specific example of why GPT 5.2’s top-down alignment is bad?

Okay, let’s take GPT 5.2. So if I’m understanding correctly, you feel that AI alignment cannot be top-down, but you think that today, OpenAI’s alignment for the consumer-facing ChatGPT 5.2 is top-down, and that’s bad, right?

Okay. So looking at the top AI companies today. Let’s say OpenAI, for instance. Do you think they’re doing alignment wrong?

Okay, fair enough. So let’s move on to this topic of AI alignment. Last month, you wrote a piece on sixpack.care saying AI alignment cannot be top-down. Explain what you mean by that.

Just poking at your own mental model here… your position is there’s also a very significant chance that it doesn’t happen like that, right? If it were to happen tomorrow, there’s still a very significant chance that it just goes bad, correct?

Well, to clarify your mental model, though, what if it literally happened tomorrow? Don’t you think that’s not enough time? Don’t you think we’re screwed if it happens tomorrow?

Yes, but the AIs that they’re building don’t, right? Especially after some recursive iterations.

But when you look at the AI companies today, are they meeting your standards for alignment? Because I think we’re gonna get to that. I think you’re saying that they’re not. We’re not on track today to have that kind of a good foom.

And you also agree with me that we’re not that confident that foom isn’t coming in six months. So if you just combine together all those propositions, don’t you get kind of like an “oh, shit” moment? There’s a big P(Doom). There’s a very significant P(Doom).

We’re going to get AIs that are much more powerful than the company SpaceX is today. So you agree with me on that, and you agree with me that we have a ways to go on getting the steering technology right, on solving the alignment problem. We have a ways ...

No, I hear you, but I’m just drawing an implication of some of the stuff you told me. You told me that you agree that a foom scenario is plausible. You and I both agree on that, and the intelligence ceiling is really high, and we’re going to get AIs ...

If you and I both knew for a fact that it was coming in six months, so like June 2026, if that’s when it was coming, wouldn’t you be way less optimistic that all the pieces would be in place for controlling that singularity if it was that soon?

It’s fair enough. Let me ask you the timelines question this way. You agree that we’re both in a very foggy headspace in terms of when exactly the singularity is coming. I agree to that. I agree I have a probability distribution where no one particular year has more than ...

So it’d be really great if we can solve social media curation or feed algorithms in the next year, and that’ll buy us a whole six years, maybe, to solve everything else before the singularity.

Nope, we suck at that, but in five years, we’re suddenly going to be so good at having all these things… You know, in five years, we’re probably still going to have the Facebook timeline, the Instagram timeline, still be engagement bait. I think it’s gonna take longer than five ...

You’re right, it’s an easier case to make, and yet the US hasn’t even gotten on board. So the reason I’m bringing up this clown show of how we handled the aftermath of COVID is because we have seven years left to get our act together before the freaking singularity, ...

Right, and it can. Right?

It’s still pretty unclear how we’re going to respond next time. We haven’t come out and made a statement, for example, on challenge trials… the idea that you can test a new vaccine, you can give people the virus. You’re still not allowed to give people the virus and test ...

Yeah, yeah. Well, just what I’m getting at when I’m saying, look at the US from 2020 to 2025… yeah, we have a few more people researching how to get mRNA vaccines online, but it’s not like we improved the CDC that much. It’s not like we actually invested that ...

I agree we are, because we know about the mRNA pipeline, but have we actually applied much of our learned lessons in the last five years, or no?

Yeah, well, okay, let’s take the COVID example. Because you’ve sounded optimistic the way you said, “Hey, we’ve learned from COVID,” but if you look at the United States, for instance, 2020 all the way to 2025, are we that much more prepared for pandemics compared to 2020?

Humanity is going to have to get so much better in the next seven years in order to play out your vision, because then it’s going to be too late to play out your vision, correct?

But the reason I’m bringing this up, the reason I think it’s worth looking at timelines, is because you have all this optimism that humanity is going to get its act together and coordinate and do the six pack of care. All these principles are going to be instilled, but ...