00:08:39.860Yeah. Maybe just to back up and so the listeners, you'll, you'll see it in the film if you go see it, but just understand the structure of the film. So the film kind of takes you on a tour of first, the people who are focused on all the things that could go wrong. And so this is the risk folks that I don't like using the term doomers because I think it reifies something that's not really healthy. You know, as someone who's worried about the risk of a nuclear power plant, a doomer, no, they're a safety person who cares about the nuclear power plant, not melting down.
00:09:04.860Yeah. Doomer is a term of disparagement launched by the people who don't share these fears.
00:09:09.440That's right. That's right. So let's not reify that. So the first film, I mean,
00:09:13.000the first section of the film is really focusing on those folks and their concerns.
00:09:16.460And it's really devastating for the director. And the story in the conceit of the film is that the
00:09:20.180director is having a baby. And so he's asking all of these people in AI, is now a good time to have
00:09:24.680a kid? And I think that humanizes the question of what is the future we're heading towards?
00:09:29.520Because in an abstract sense, it's not that motivating. When I think about me and my kids,
00:09:32.700it anchors this discussion about AI in terms of the things that people most care about,
00:09:36.680which is their family. So then the film, after the director sort of is confronted by all this
00:09:41.080and he gets overwhelmed and he kind of freaks out to his wife thinking, oh my God, I don't know what
00:09:45.740to do. And she says, you have to go find hope. And so he turns around and he goes out and he
00:09:49.260talks to all of the AI optimists. So this is Peter Diamandis. This is, you know, Guillaume Verdun,
00:09:54.060who's Beth Jezos, otherwise known as online, basically the tech accelerationists and people
00:09:58.920who think that our biggest risk is not going fast enough. Think of all the people with cancer or all
00:10:03.780the people whose lives that we won't be able to save if we don't make AI faster than we're making
00:10:08.640it right now. My reaction, I think going sort of a step back, there's a thing in AI that we have to
00:10:16.840acknowledge there's an asymmetry. The upsides don't prevent the downsides. The downsides can
00:10:22.120undermine a world that can sustain the upsides. So for example, the cancer drugs can't prevent
00:10:28.020a new biological pathogen that's designed to wipe out humanity. But the biological pathogen
00:10:32.500that can wipe out humanity undermines a world in which cancer drugs are relevant at all.
00:10:36.800AI generating GDP growth of 10, 15% because it's automating all science, all technology
00:10:42.340development, all military development, automating abundance sounds great. But if the same AI that
00:10:47.540can do that also generates cyber weapons that can take down the entire financial system,
00:10:52.140which one of those things matter more? 15% GDP growth or the thing that can undermine the
00:10:55.960basis of money and GDP at all. So it's very important. The film doesn't actually make this
00:11:00.660point. And it's one of the critical things that people do need to get because in order to be
00:11:04.960optimistic, you have to actually mitigate the things that can go wrong. And I feel like AI
00:11:09.320is presenting us with essentially a maturity test. It's almost like the marshmallow test
00:11:13.560in psychology, where if you wait and you actually mitigate the downsides, then you get the actual
00:11:19.320two marshmallows on the other side of the genuine benefits of AI. But if you sort of race to get
00:11:24.340the one marshmallow now and don't mitigate the downsides, then you get the downsides.
00:11:27.940And I think that is not in the film, but is critical for people to get.
00:11:31.520Yeah. Yeah. So then what do you make of the people who have all the facts in their heads,
00:11:35.860but they're not worried or claim to be not worried about quite literally anything?
00:11:41.880Yeah. Well, personally, I think there's an intellectual dishonesty there. And I'm sure
00:11:47.000in past conversations, you and I have had Sam over the years.
00:11:49.560But there's an interesting case here. So take someone who has finally had their religious epiphany here, but for the longest time didn't. And this is literally the most informed person on earth, Jeffrey Hinton. How do you explain that these problems weren't obvious to him years ago?
00:12:10.960Oh, so you're saying for Hinton that he had an awakening or something?
00:12:13.580Yeah, so he was somebody who didn't give really any credence to concerns about alignment that I'm aware of for years and years and years, as he was quite literally the father of this technology.
00:12:32.260Why? I mean, it's not that he got more information, really. So how do you explain his journey?
00:12:37.480Well, so I don't know his particular journey. You might just know more about what his awakening moment was. So I can't, I can't really speak to that.
00:12:44.600I think it was just that he, I mean, this has always been a non sequitur from my point of view, but it was just his sense. I think this is what he said publicly that the time horizon suddenly collapsed. We just suddenly made much more progress than anyone was expecting.
00:12:57.700Well, that generally has been one of the things. I mean, it's the thing that caused those AI
00:13:01.220engineers, kind of the Oppenheimers in January, 2023 to reach out to me. And that's what it felt
00:13:05.700like. It's like you were getting calls from people inside this thing called the Manhattan Project
00:13:09.300before I knew what the Manhattan Project was. Right. Because to be clear, I actually went to
00:13:13.700the, I went early on to like an effective altruism global conference. I was not an EA,
00:13:18.600but I happened to go to the conference in like 2015. And I was actually frustrated because I
00:13:22.440felt like the EA community was obsessed with this virtual risk called AI that I didn't take
00:13:26.440seriously back at the time because we were nowhere close to those capabilities and i was like there's
00:13:31.120a big runaway ai here right now that went rogue it's maximizing for a narrow goal at the expense
00:13:35.220of the whole and it's called social media and ea is completely oblivious to it and isn't focused
00:13:39.120on it but then i was really wrong later when ai capabilities really just made a huge amount of
00:13:44.380progress and that's again when we got the calls from people in the lab so i think it was the jump
00:13:47.700of just suddenly hey i think gpt4 will like pass the bar exam pass the mcats like that's suddenly
00:13:53.400a new level of AI that we just didn't have before. Yeah. Yeah. Yeah. I still have no theory of mind
00:13:59.200for the people who are not worried now about anything. Everything from the comparatively
00:14:05.020benign, like just economic dislocation and wealth concentration that's unsustainable politically
00:14:09.820to the genuine concerns about alignment, that we could build something that we are now trying to
00:14:15.140negotiate with that has more power than we have and we can't take the power back. I mean, to be
00:14:19.600fair, just to say it bluntly, I think some of them are lying. I think some of them actually do
00:14:23.240are building bunkers right now. Just let's just say it. They're building bunkers and they
00:14:28.440simultaneously say everything. There's all these amazing things we're going to get.
00:14:32.540They sort of wash over with their hands. They kind of push away the idea that there's going
00:14:37.280to be all this disruption in the middle, middle time. And they're kind of focused on the long
00:14:40.300term. Like after we make it through this basic horrible disruption and maybe revolutions,
00:14:44.060there's going to be some other side of this, which will be the most abundant time in human history.
00:14:47.900Right. People like this, they often point to the graph of global GDP, where if you look and during 1945, you barely get a little blip where it goes down for a moment and then it goes straight back up. Right. And it's that kind of psychology. There's also the psychology of Upton Sinclair that you can't get someone to question something that their salary depends on not seeing. Yeah. And so if your business model is selling optimism and selling hope and selling everything's going to be great, you're obligately not able to speak about the risks.
00:15:15.980But I think this is the thing we should be watching out for is just incentives are the problem with the world. Incentives that allow non-honest speech to be the public understanding that we need to operate on. Because we just need objective sense-making and not incentivize sense-making.
00:15:29.180And we know that some of the principal people doing this work, people like Sam Altman and Elon Musk, were people who were at first as worried as anyone.
00:20:29.340Yeah. This is so critical because I think if you view AI as just another technology that confers
00:20:34.640power, it's a tool, you pick up that tool and use it like any other, you end up in one world.
00:20:38.500But if you see that AI is the first technology that thinks and makes its own decisions and is generating hundreds of thousands of words of strategic reasoning when you ask it a basic question or like how to code, suddenly you end up in a different world.
00:20:50.020So let's talk about some of these examples of AI uncontrollability.
00:20:53.920So in the film, they reference this example that many people have heard about by now of the anthropic blackmail example.
00:20:59.880This is a simulated company email where in the simulated fictional company, they say in the emails to each other, we're going to shut down and replace this AI model. And then later in that company email, there's an email between the executive at the company and an employee. And the AI spontaneously comes up with the strategy that it needs to blackmail that employee at Anthropic in order to protect itself, to keep itself alive.
00:21:24.160at first people thought well this is um you know just one bug and one ai model but then they tested
00:21:30.660all the other ai models from deep seek chat gpt gemini grok etc and they all do the blackmail
00:21:37.020behavior between 79 and 96 percent of the time yeah amazing yeah there's this kind of moment
00:21:43.320where there's like cue the nervous laughter like it's and yet if you actually send this to people
00:21:48.140who are at the white house i think there's a disregard for this people just say well you're
00:21:51.700coaxing the model. You're, you're, you're getting it to do this. You're kind of trying to, you're
00:21:55.540trying to put it in a situation where of course you're going to like keep tuning the variables
00:21:58.700until you get it to blackmail. So I have some updates since then. Anthropic trained another
00:22:04.560model. They were able to train the blackmail behavior down by quite a lot. So it doesn't do
00:22:09.000this behavior in this simulated environment. That's the good news. The bad news is that the
00:22:13.300AI models are now situationally aware of when they're being tested and they're now altering
00:22:18.660their behavior way more right right yeah that that's uh strikes me as genuinely sinister yes
00:22:24.640i think we have a hard time modeling because all of this abstract i mean i'm just thinking
00:22:28.300about your listeners and it's like this just sound like where you don't have you know back
00:22:32.640to e.l wilson the fundamental problem of humanity is i have a paleolithic brain we have medieval
00:22:36.420institutions and god-like technology and the only experience you have with your brain with regard to
00:22:41.440ai is this blinking cursor that tells you why your washing machine is broken that's different than
00:22:46.960this blackmail example that sounds abstract and that you don't actually experience that side of
00:22:50.780AI. But the thing that, I mean, again, I've thought about this enough in the vein in which
00:22:57.080I've thought about it for now at least 10 years, where it was obvious to me, I don't consider
00:23:02.220myself especially close to the intellectual underpinnings of any of this technology, right?
00:23:06.720I'm just a consumer of the news on some level with respect to AI. But you were right and reasoned
00:23:11.420about it philosophically and was able to get to the right conclusions. It was just so obvious that
00:23:13.920The moment you can see that intelligence is not substrate dependent, that we're going to build actual intelligence in our machines, given what intelligence is, you should expect things like deception and manipulation and the formation of instrumental goals that you can't foresee.
00:23:30.680And certainly when you're, when you're imagining building something that is smarter than we are.
00:23:41.480So that every time, I mean, just how would this conversation go if every time I uttered a sentence, you functionally had two weeks to decide on your next sentence?
00:23:54.740Long before you get superhuman AI, you just get super speed.
00:23:57.460Yeah, so speed alone is enough to just completely outclass you. And intelligence, you have to envision this as a relationship to a mind that is autonomous. And then you add things like, you know, recursive self-improvement and all that. And then all of a sudden, you know, we're in, you know, some dystopian science fiction if this is not perfectly aligned.
00:24:18.620That's right. Let's make sure we add just another example because there's a recent example from just three weeks ago. Alibaba, the Chinese AI company, was training an AI model and then totally in a different side of the company, their security team noticed a bunch of network activity, like a flurry of network activity, like what the hell is going on here?
00:24:37.200And it turned out that in training, midway through training, not deployment, in training, the AI model had basically set up a secret communication channel with the outside world and then had started to independently start mining for cryptocurrency.
00:24:50.860Now, this time you cannot claim that someone coaxed the model to do this.
00:24:56.060This is spontaneous instrumental goals of the best way to do any goal is to acquire more power and resources so you have the ongoing ability to achieve those goals.
00:25:02.960And it went to decide to acquire cryptocurrency.
00:25:05.240right now if you're a chinese military general and you hear this example like how do you feel
00:25:11.100as a mammal like you feel the same way that any other goddamn mammal feels hearing this example
00:25:14.960if you're a u.s military general and you hear this example it's terrifying as a human being
00:25:21.000so there's like a good news in this for me which is that i think people just literally don't know
00:25:25.560these examples they just don't know like what percentage of the world's leaders do you think
00:25:30.080are aware of this alibaba spontaneously mining cryptocurrency example like if you had to guess
00:25:34.780Oh, I would think it's minuscule, but I mean, there's also, it does seem like there is still a barrier to internalizing any of these examples with the appropriate emotional response.
00:25:47.780It's like, I mean, they're, again, this is, I come back to the way that struck me the first time I started thinking about it 10 years ago in my TED talk on this topic in 2016.
00:25:56.860I remember starting with the problem, which is as worried as I can be about this for the next 18 minutes, all of this is fun to think about.
00:26:05.940Like, this is not the same thing as being told that actually your landscape has been contaminated by radioactive waste, you know, and you can't live there for the next 10,000 years.
00:29:28.020and in some cases, even psychopathic people, right?
00:29:30.240The system has selected for the psychopathic people.
00:29:32.900with some of the people who are in charge in our own case, but leaving that aside,0.96
00:29:37.300we're in an arms race with China, right? We're in an arms race with, I mean, I guess China's0.97
00:29:41.440the most plausible, but who knows who else, but we're probably in an arms race with Russia. I0.94
00:29:45.560don't know where Russia is on this, but, and when you think of the prospect of any, you know,
00:29:51.460authoritarian slash totalitarian regime, getting this technology first in a, in what
00:29:58.100will look like something like a winner take all scenario. If, if there really is a
00:30:02.900binary, you know, step function into super intelligence that, you know, to be, you know,
00:30:09.200two months ahead of the competition is to basically win the world. And we could be in
00:30:12.840some situation like that in the event that that just doesn't destroy everything, right? If it
00:30:18.040just actually confers real power, right? Because it's sufficiently aligned with the interests of
00:30:22.800whoever develops it. That is so compelling that we just, we cannot lose to China above all here.
00:30:28.120Certainly when you're talking about, you know, autonomous military technology or,
00:30:31.040uh, you know, anything that would be deployable in our own defense or offense, right? You know,
00:30:36.820cyber security. Sure. Like we, we, we can't be behind. So how do we, how do we become slow and
00:30:42.340careful under those conditions? Right. But then what are the chances that that super intelligent
00:30:47.640AI that gives us that dominance, we will control? Right. So that's no, no, literally what are the
00:30:52.120chances? Well, this is a point you've made. Uh, I don't know if you make it in the film,
00:30:54.980but I've heard you make it, which is, you know, we, we were first with social media, right? We,
00:31:00.240You know, like we were, if you look at that as an arms race that we won, correct. What exactly did we win? Exactly. We, that winning that arms race to invent essentially like a psychological manipulation weapon, a mass behavior modification engine machine with AI. We built that first, but then we didn't govern it well. So it's like a psychological bazooka that we flipped around and blew off our own brain. And so what that shows you is that we're not actually in a race for who has the most power.
00:31:25.780We're in a race for who is better at steering, applying, and governing that power in ways that
00:31:30.860are society strengthening. That is what we're actually in a race for. Because if we actually
00:31:35.160beat China to a AI bazooka that we literally don't know how to control, and we're not on track to0.65
00:31:40.800know how to control, and all the evidence shows that it has more self-awareness of when it's
00:31:45.740being tested, not less. It is better at cyber hacking, not less. It is better and does it more
00:31:50.800often these kinds of self, you know, preserving behaviors. If we're not on track and we're also
00:31:55.960going faster, like the conditions in which we would be on track to control it would be the
00:31:59.960ones that were going slow and steady. But we're doing the opposite of those conditions because
00:32:03.060of the race dynamic. So there's just this kind of psychological confusion here, which is we're not
00:32:08.380going to win this race. In the race between the U.S. and China, AI will win. There's a metaphor
00:32:12.420that our mutual friend Yuval Harari, who's the author of Sapiens, has here, which I guess in the
00:32:17.160the post Roman period of the British empire, it was very weakened and they were getting attacked
00:32:22.480from the Scots and the Picts in the North, basically, you know, pre-historical Scotland
00:32:28.080and Ireland and those civilizations. And they were very weak. And they said, what are we going to do?
00:32:33.220They had this idea. Well, why don't we go off and hire this bad-ass group of mercenaries called
00:32:37.020the Saxons? Because those Saxons are super powerful. And if we get the Saxons to fight0.92
00:32:42.240our wars for us, then we'll win. And of course we know the history of how that went. We got the0.99
00:32:46.520Anglo-Saxon empire, except in this metaphor, AI is the Saxons. Except we won't get a merger between
00:32:51.980the human AI empire. We will get the AI empire. This makes me think of all these guys in their
00:32:58.500bunkers who have hired Navy SEALs to protect them for the end of the world. They're going to control
00:33:04.400their Navy SEALs until the end of time. Exactly. But this is insanity. So the main point here is0.52
00:33:09.660that there's kind of an attractor that's driving all of this right now, which is this arms race
00:33:13.760dynamic, under this false illusion that we have to beat China, but we're not examining the logic0.52
00:33:17.820of what are we beating them to. We're beating them to something that we don't know how to control,0.97
00:33:21.720and we are not on track to control. And then you get people like Elon saying, in this weird,
00:33:25.760and I'm curious what you make of his psychology, but saying in public interviews, I think it was
00:33:28.420in the Cannes Film Festival or something in France, and he said, I decided I'd rather be
00:33:31.780around to see it than to not. It's kind of this surrender. It's kind of this death wish. It's kind
00:33:36.000of like, I can't stop it, so I decided I'd rather be there to have built it and have my God be the
00:33:41.500thing that took over. This actually is a fundamental thing that we should double click on
00:33:45.800for a second, which is the unique thing about AI game theory that's different than nuclear game
00:33:50.480theory, which is that the omni lose-lose scenario from nuclear game theory is like, I know as a
00:33:56.940mammal that you also don't want to annihilate all life on planet earth. And the fact that I know
00:34:01.960that about you without even talking to you means that there's some element of trustworthiness that
00:34:05.980we will try to coordinate to something else because we agree on some implicit level there's
00:34:10.700an omni lose-lose thing that's worth avoiding. Here's the problem with AI. If I start by
00:34:15.780believing that it's inevitable and nothing can stop it, then if I'm the one who built the suicide
00:34:21.440machine, I'm not an evil person because I'm only doing something that would have been done anyway.
00:34:27.180So I have an ethical off-ramp in that decision. And the second part is, unlike if you literally
00:34:32.420made it like a matrix where you just get the point scores of, you know, you get negative infinity if
00:34:36.580we get nuclear war in the nuclear scenario with AI. Let's say we're in this race and the DeepSeek
00:34:41.900CEO is there and the Elon's there and Sam's there and they're racing to do it. They actually all
00:34:46.120believe it could wipe out humanity. But if they raced and got there first, then think about the
00:34:51.560scenario. Humanity is wiped out, but there now exists an AI that speaks Chinese instead of0.89
00:34:56.900English or has the DeepSeek CEO's DNA rather than Elon's DNA. The end of the world has your logo on
00:35:02.980That's right. Exactly. Good. Well said. So the end of the world has your DNA or your logo on it. And I want people to get this because if people got this, they would see that there might be an implicit way that people might think that like when push comes to shove, you know, cooler minds will prevail because you can trust that the people at the top will like do whatever it takes to steer away from this and will like steer away in time.
00:35:22.600But what I want people to get is you can't trust that because these people actually subconsciously, I think there's psychological damage here that I think they subconsciously have pre accepted this kind of end of the world and end of their life. And that if they got to be the one who built the digital God that literally was replacing humanity in some legacy in some world, I don't know whose history book that exists or someone, anyone's conscious going to read that, but they got to go down in history in that way. And what that does is it should motivate the rest of the 8 billion people on planet earth to say, I'm sorry to swear, but just fuck that.
00:35:52.600We don't want that. If you do not, if you want your children to live and you care about the world as it exists and you love the things that are sacred about life and you're connected to something, that is at risk with this small number of people who are racing to this negative outcome.
00:36:06.140Well, a lot of these guys seem to have, um, had their formative educational experiences reading science fiction. I mean, it's like you read a lot of science fiction, you read a little Ayn Rand and you're self-taught in basically everything else.
00:36:21.960And you, um, to my eye, you form a very weird set of kind of ethical weights, you're just, you're just, you're just not enough of the best parts of culture have, uh, gotten into your head such that you can actually, um, come to a real understanding of what human life is good for.
00:36:44.100I mean, you literally meet people who are agnostic as to whether or not it would be a bad thing if we all got destroyed and ground up in this new machinery and our descendants were, were robots where, where in the consciousness may or may not exist.
00:36:59.600And they're like totally kind of like, maybe that's, that's sort of an interesting way to end this movie.
00:37:03.500I mean, you get a semblance of that when Peter Thiel is asked the question by Ross Douthat in the New York Times, should the human species endure?
00:37:10.180It's a real, it's a real stutters for 17 seconds.
00:42:54.680I mean, a Chernobyl scale event might be the best case scenario at this point.
00:43:01.060Something that gets everyone's attention in a transnational way.1.00
00:43:04.900something that actually brings china and the america to the table with you know ashen faces
00:43:11.700wondering how they can collaborate to move the final yards into the end zone safely yeah you
00:43:18.180need something it's hard to imagine what is going to solve this coordination problem short of
00:43:23.380something that's terrifying yeah i mean i i so if i could there already are as you i'm sure are well
00:43:29.940aware these international dialogues on AI safety, track two dialogues between U.S. and Chinese
00:43:34.900researchers, but they're happening at a low level. They're not blessed by the tops of both countries.
00:43:39.660There's not a regime of regulation, certainly on our side, that is going to force anyone to do
00:43:45.280anything. No. And I think, I mean, actually, to be fair, I think China actually is quite concerned
00:43:50.200about these. To be clear, the Chinese Communist Party does not want to lose control. That is
00:43:54.280like their number one value. So they do not want to let, and they will not let AI run amok. They
00:43:59.180will probably regulate in time, but they're probably looking at us and saying, what are you
00:44:03.460doing? We're the scary ones in this relationship. And notice that they lose if we screw it up and
00:44:07.940we lose if they screw it up. So again, forget kumbaya, we need coordination and a treaty is
00:44:12.420going to happen. No, even if you don't do that, you just come from pure self-interest. From pure
00:44:16.520self-interest, we can't afford to get this wrong. And as Aza, my co-founder says in the film,
00:44:20.940the AI doc, this is essentially the last mistake we ever get to make. So let's not make it.
00:44:25.220So what are you expecting in the near term? Let's leave concerns about alignment aside, unless you think we're going to plunge into superintelligence in the next 12 months. What will you be unsurprised to see in the next year or two? And what are you most worried about?
00:44:43.700I mean, we're furthering down the trajectory of mass joblessness, which maybe we should just
00:44:50.120briefly articulate why, you know, there's always this narrative. It's just important to debunk
00:44:54.180these common myths, which are essentially forms of motivated reasoning and looking for comfort.
00:44:58.220We're comfort seeking, not truth seeking. So one of the ways we're comfort seeking is like,
00:45:02.280hey, there's a, there's a, you know, narrative out there that 200 years ago, all of us were farmers
00:45:06.300and now only 2% of whatever the population is a farmer. And we always find something new to do.
00:45:11.820the tractor came along. We had the elevator man. We used to have the elevator man. Now we have the
00:45:15.520automated elevator. We used to have bank tellers, have automated teller machines. Jeff Hinton was
00:45:19.340wrong about radiology, blah, blah, blah, blah, blah. What's different about AI is that this kind
00:45:24.600of artificial general intelligence is that it will automate all forms of human cognitive labor
00:45:30.020all at the same time, or roughly progressing on that trajectory. You still get jaggedness,
00:45:34.680which is the term in the field of slightly more progress, for example, on programming than you do
00:45:39.600on, I don't know, complicated social science issues or something like that. But what that
00:45:44.320means is, you know, Attractor didn't automate finance, marketing, consulting, you know,
00:45:49.620programming all at the same time. AI does do that. And who's going to retrain faster,
00:45:54.240the humans or the AIs? So I just want to say that because it's worth debunking this idea that
00:45:59.920humans are always going to find something else to do. We'll do something else. And it's great
00:46:03.080for people to retrain and learn to vibe code. But AI is using all that training data from all
00:46:06.980the people vibe coding and using that to make the better the better system yeah and you know one of
00:46:13.000the most popular jobs actually we're in la right now and one of the most popular jobs in la that
00:46:16.900was covered in the la times recently i'm sure you saw the story they call them arm farms no i didn't
00:46:21.840see this is basically someone straps a gopro to their top of their head and then they just fold
00:46:27.580laundry or do tasks with their hands oh so that's robots are learning how to do that it's right so
00:46:32.820essentially the number one job in the world would be training our replacement so essentially we all
00:46:36.780have the job of coffin builders. We're essentially our number one job is we're in the coffin making
00:46:40.820industry to replace us with AIs that will do that job more effectively and for cheaper in the future.
00:46:46.060If we don't want that, and obviously there's going to be things that we still value in this
00:46:50.480new world that are human to human interaction, a nurse, we don't want a robot nurse. We want a
00:46:54.860human nurse and we can definitely train more nurses. And so I don't want to say that it's
00:46:58.060a hundred percent of all automation is going to happen, but the goal of these companies is not to
00:47:02.560augment human work. This is so critical for people to get. You know, you heard J.D. Vance say in the
00:47:08.240speech when he first came into office at the first AI summit in France, and he said, you know, AI will
00:47:13.620augment the American worker. It's going to support workers to be more productive. But what is the
00:47:18.020business model of OpenAI and Anthropic and these other companies? If we're again using this Charlie
00:47:23.300Munger incentive framework to predict their choices, like what is their business model? And
00:47:27.380people say, oh, okay, there I am using ChatTPT. What's their business model? How do they make
00:47:30.760money. Oh, I pay them 20 bucks a month for the subscription. That must be how they're going to
00:47:34.520make money. But that's actually not what it is because the 20 bucks a month, if everybody paid
00:47:38.640it, that would not make up all the money and debt that they've taken on as a company. It wouldn't
00:47:42.740work. Okay. So that's now. So what's the next one? What about advertising? Let's do the Google thing.
00:47:46.520Let's do mass advertising for all these AI models embedded in the results. We're going to have,
00:47:51.320this is going to be the new search. Search is one of the most profitable business models in the
00:47:54.540world. Maybe that will do it, but that doesn't also make back the amount of money these companies
00:47:58.480have taken on. The only thing that makes back the amount of money these companies have taken on
00:48:02.460is to replace all human economic labor to take over the $50 trillion labor economy. That is the
00:48:09.540price. It's artificial general intelligence, which means replacing human work, not augmenting human
00:48:15.060work. It's just so critical for people to get that because again, this gets you the sort of
00:48:18.840sealing the exits on why we're heading to an anti-human future. That's my goal here. My goal
00:48:22.720here is just if you can see the anti-human future clearly, if everybody in the world got that,
00:48:26.200I honestly think, Sam, if literally if every human in the world got that, I do think that we would steer to do something else.
00:48:31.300Well, it all falls out of what we mean by the concept of general intelligence, right?
00:48:37.160So once you admit that we're building something that by definition is more intelligent than we are, right?
00:48:44.160And any increment of progress, provided we just keep making that progress, is eventually going to deliver that result.
00:48:52.200Leaving aside the alignment problem, let's say it's just perfectly aligned, right?
00:49:24.680So like in situate, I'm not even sure nurses in the end survive contact with this principle, but for those things where we are always going to want the human in the loop, right. Or the human to be the origin of the product, whether it's, you know, music or novels or, you know, stage plays, maybe, I don't think we're, maybe we're never going to want to see robots on stage acting Shakespeare.
00:49:48.360We're never going to want to see, you know, robots in the NBA because it's just, we just want to see what the best people can do in the NBA.
00:49:55.840But still, you're talking about, you know, 1% of the human employment there.
00:50:02.280So there are jobs that will be canceled and they'll be canceled for all time in the same way that being the best chess player in any room has been canceled for all time.
00:56:07.760I think that there should be common knowledge. I think if everybody at that class knew about these examples, even without a formal agreement or treaty, we would do something else. And you can do that even under conditions of maximum geopolitical rivalry. So as an example is in the 1960s, India and Pakistan were in a shooting war, and they still were able to do the Indus Water Treaty, which was the existential safety of their shared water supply, which lasted over 60 years.
00:56:31.800So the point is you can be under maximum geopolitical competition and even active conflict while collaborating on existential safety. We just have to include AI in our definition and domain of what existential safety is. The Soviet Union, the United States, also under maximum competition in the Cold War, collaborated on distributing smallpox vaccines. Again, so there are examples of this throughout history, even under maximum rivalry.
00:56:53.920So that's number two, is we need some kind of international limits.
00:56:56.480And at the very least, we need common knowledge of what would constitute those guardrails.
00:57:00.360The one big one is you should not have closed loop recursive self-improvement, meaning someone
00:57:05.820hits a button and the AI runs off and does all the experiments and rewrites itself a
01:00:57.920Is it true to say that there's basically no regulation at this point?
01:01:02.400I think it's incredibly minimal. There's like the Take It Down Act, which is around sexualized deep fakes, and you're obligated to take those down. There's just a couple limited examples, but almost no regulation. I mean, as they say in the film, Connor Leahy from Conjecture will say there is more regulation on a sandwich, on making a sandwich in New York City than there is in building potentially world-ending AGI.
01:01:20.400yeah but that should inspire people like we there's everyone's on the same team no one wants
01:01:25.680an anti-human future no one wants no ability to meet make their ends meet and have their kids you0.90
01:01:30.600know fucked up by ai that's screwing them with ai psychosis that takes away their political power0.88
01:01:35.620so they don't have any voice in the future like everyone wants the same thing and i know it doesn't0.82
01:01:40.160seem that way right now but especially when you add in there the rogue ai examples of it super
01:01:45.860intelligent you know hacking systems and we don't know how to control and it's mining for
01:01:49.140cryptocurrency, again, every country in the world has the same interest. Every human has the same
01:01:54.140interest. We're just not seeing the invisible consensus. And one other point of optimism is
01:01:59.260Future of Life Institute, which I know you know, Max and the good people over there who've done
01:02:03.320amazing work on this. They brought together a hundred and something groups to New Orleans
01:02:09.000earlier this year, and they came up with something called the Pro-Human AI Declaration. And they
01:02:13.440basically had 46 groups sign on to five basic principles of what we want. And it's basic stuff
01:02:18.740like human agency and what kind of groups are we talking about? Yeah. So this, this pro-human AI
01:02:22.700statement, they actually call it the, the B2B coalition or the Bernie to Bannon coalition,
01:02:27.740because everyone from Bernie Sanders to Steve Bannon like agrees on this. These are 46 groups
01:02:32.460like the, you know, church groups, uh, evangelical groups, uh, Institute for family studies, AI
01:02:37.140safety groups, many, many different groups across the political spectrum, across the religious
01:02:40.720spectrum. And they all agree on these five key principles. One, keeping humans in charge to
01:02:45.820two, avoiding concentration of power, three, protecting the human experience from like AI
01:02:50.060manipulation, psychological hacking, four, human agency and liberty, like no AI-based surveillance,
01:02:56.260and five, responsibility and accountability for AI companies, things like liability, duty of care,
01:03:01.460et cetera. So there's actual policies that are behind that. But the point is that this is
01:03:05.520something that we all agree on. Again, there's actually much more consensus and agreement than
01:03:09.300most people think. I think right now, 57% of Americans in a recent NBC News poll say that
01:03:15.660the risks of AI currently outweigh the benefits of AI and that AI is less popular. I think it's
01:03:22.200at 27% of the population has positive feelings about AI in this country. So now I know that
01:03:28.660someone like David Sachs listening to this says, if you look at China, people are super positive
01:03:32.700and optimistic about AI. And this is why we're going to lose the race is that there's all these,
01:03:35.720this positive excitement about AI. So they're going to deploy it and then we're going to lose.
01:03:39.040But I don't think that what you should interpret is that we're wrong and just
01:03:43.160misassessing the dangers of AI, I think that we have not collectively yet woken up to the dangers
01:03:48.800of AI. And again, we can actually accelerate all the positive narrow use cases where it's actually
01:03:54.020improving education, actually improving medicine, actually improving and optimizing energy grids
01:03:58.440and things like that that are not about building super intelligent, general autonomous gods that
01:04:02.160we don't know how to control. So there's a way to accelerate the kind of defensive applications of
01:04:06.600AI and narrow AI without accelerating general and autonomous AIs that we don't know how to control.
01:04:11.680so there there is a way through this but it's like it requires you have as i said in the trailer
01:04:16.560of the film it's like we have to be the wisest and most mature version of ourselves and by the
01:04:20.360way i i'm realizing especially talking to you sam that this is the hardest problem that we've ever
01:04:26.080faced as a species so it's i'm not saying and then some of the pieces people into false optimism yeah
01:04:30.340i mean the things that worry me the most are the people i mean among among the things that worry me
01:04:35.380the most one is the the testimony of the people again who are close enough to the technology to
01:04:42.260be totally credible who won't concede any of these fears right i mean so it's um it's that they do
01:04:49.300and they don't it's like it's weird you'll hear sam talk about the risk he just said didn't interview
01:04:52.600in the last couple days and he talked about the risks of a major cyber event this year yeah he's
01:04:56.680unusual i mean he's an unusual voice in that he will if you i haven't seen him lately ask this
01:05:02.080question but you know last time i saw him asked point blank about the alignment problem he totally
01:05:07.340concedes that it's a problem right so and so like there's there's there's the way in which this could
01:05:12.600go completely off the rails and it's you know this is intrinsically dangerous if not aligned
01:05:18.480i just want to move that from could to will like we are currently not on track like if you if you
01:05:24.260just let it run everything right now we would it would not end well right yeah yeah i mean there's
01:05:29.360just probabilistically, you have to imagine there are more ways to build super intelligent AI that
01:05:36.640are unaligned than aligned, right? So if we haven't figured out the principle by which we would align
01:05:41.920it, the idea that we're going to do it by chance seems far-fetched. That's right. I think people
01:05:46.520like Stuart Russell, again, who wrote the textbook on AI, will point out that I think a nuclear
01:05:51.220reactor has something like an acceptable risk threshold of one in a million per year, meaning
01:05:57.600like there's a one in a million chance per year that you get a nuclear meltdown somewhere between
01:06:02.360that and one in 10 million i think well when you ask someone like sam altman what are the what's
01:06:05.680the probability we're we're going to destroy everything with this technology and the answer
01:06:09.580is like between 10 and 20 yeah 30 no one's saying one in a million right yeah so we just need to
01:06:15.020stop there for a second it's like i know it's easy to run by these facts but it's like let that into
01:06:19.080your nervous system yeah let that land no one wants that no one wants that right but there's
01:06:25.680this miss where it's i think so much of the issue sam is there's this crisis of human agency where
01:06:30.800you can't so when i say no one wants that i know what someone might be thinking it's like yeah but
01:06:34.460what can i do about it because the rest of the world is building it and i i don't have so i
01:06:38.500might as well join them you get this whole like weird psychology well there are five p i mean
01:06:42.240there's there's only something like you can count on one or at most two hands the key the number of
01:06:47.880people whose minds would have to change so as to solve this coordination problem at least in america
01:06:52.720we haven't, have we really tried? Like, have we really just really gotten in the room? I mean,
01:06:56.300Bretton Woods, which was the last time we had a transformative technology, the Bretton Woods
01:06:59.720conference happened after World War II to basically come up with a structure that could stabilize a
01:07:04.560global order in the presence of nuclear weapons, creating positive sum economic relations and the
01:07:08.660whole currency system, et cetera. And that was a more, I think it was a month long conference at
01:07:13.040the Mount Washington Hotel in New Hampshire with hundreds of delegates, like you work it through.
01:07:17.240We haven't even tried locking the relevant parties in a room and saying, we have to figure this out.
01:07:20.980we haven't even tried well i want to actually go back to one really quick thing this crisis of of
01:07:26.200kind of the experience of agency with respect to this problem i just want to like dwell on this
01:07:29.740point for a second we did a screening of the film in new york uh i guess it was a week ago and some
01:07:35.100we did a q a at the end of the screening someone was in the room who is a executive coach to the
01:07:41.360top executives of one of the major ai players and their response to the film was even as like a
01:07:47.000either just super senior executive or even CEO level, you talk to the people building this
01:07:52.100and they say, yeah, I agree, but what can I do? How could I steer it? I want people to take that
01:07:56.800in. It's like the people who are maybe CEO level at these companies do not experience that they
01:08:02.000have agency. There's a problem with AI where you will never locate enough agency to address this
01:08:07.540problem inside of one mammalian nervous system who's looking at this problem.
01:08:11.240right this is actually a coordination problem has represented his situation i don't know if
01:08:17.520this is honest maybe but for years he's been saying when asked you know regulate me right
01:08:24.080like yeah you know i can't do this myself yes i need to be regulated yeah and so um that's what
01:08:30.500we said in the film too that what motivated us to do this work going back to the original story of
01:08:35.480that january 2023 phone call and running around the world was we talked to people in the labs and
01:08:39.440like, you need to figure out a way to get the institutions to create guardrails to prevent
01:08:42.840this. And then, so we fly off to DC and we say like, okay, our people inside of San Francisco
01:08:46.920are telling us you need to create guardrails. And their response is like, we're dysfunctional.
01:08:51.620We can't do it until the public demand is there. And then everyone is essentially pointing a finger
01:08:55.640at someone else to say that you have to move first to make something happen. But what they
01:09:00.240all agree is there needs to be mass public pressure. And I forgot to mention that as part
01:09:05.080of the response to the film, we call it, there's kind of a movement to respond to this, and that's
01:09:10.260the human movement. I mean that in the sense that what is the size of the object that can move the
01:09:14.580default incentives of trillions of dollars advancing the most reckless outcome as fast
01:09:18.180as possible? And the answer is all of humanity saying, I don't want that anti-human future.
01:09:23.240And one thing to point out, I mean, I think it was more or less explicit at one point in this
01:09:27.300conversation, but might've gone by unnoticed, is that the alignment problem is arguably,
01:09:32.240it's the scariest problem it's this is where we ruin everything but it is fully divorceable from
01:09:38.720all these other problems which in their totality are still quite bad right so i mean we're living
01:09:44.680in a world now where if we were just simply handed by god a perfectly aligned ai super intelligence
01:09:51.480that so it's going to do exactly what we want it's never going to go rogue we don't have to
01:09:56.120The world's not going to be tiled with solar arrays and servers. It still has all of these
01:10:02.900unintended effects that we have to figure out how to mitigate. Wealth concentration,
01:10:09.420mass unemployment, the political instability of all of that in the case of alignment, but still
01:10:15.260technology that can be maliciously used, the bad actor problem. I mean, if you can cure cancer,
01:10:19.860you can also spread some heinous virus that you've synthesized. So we have an immense problem
01:10:27.820to solve, even if there was no concern about anything going rogue on us. If you literally
01:10:33.160just paused progress right now, this would still be the fastest technology impact, comprehensive
01:10:40.160set of impacts that we've probably ever experienced. Just metabolizing the impact of what we already
01:10:44.720have, it would already be the fastest rollout we've ever had. And by the way, just to, one of
01:10:49.400things about doing this work and being located in Silicon Valley is we talk to people at the labs
01:10:53.400and you always have to be confidential and protect people's sources. But a stat that I have heard
01:10:57.920is that if you were to pull people at Anthropic right now, that their preference, the people who
01:11:02.700are closest to this technology, they would say that 20% of the staff would say, pause right now,
01:11:08.340don't build more. That's just a relevant piece of information. Imagine 20% of the Manhattan
01:11:12.520Project just said, hey, we're building a nuclear weapon. We probably should stop right now. 20%
01:11:16.520said that. You have to ask are the rest, what are the rest beliefs? But I just think people need to
01:11:21.940get that it's like, there's, as you said, there's so many problems that this is just introducing
01:11:26.520across the board that we'd be better off having this technology rollout happen at a speed at which
01:11:31.840our institutions and our public and our culture can respond to it. It's almost like Y2K, except
01:11:36.580it's like Y2AI. Like there's suddenly all these new vulnerabilities across our society, but it's
01:11:41.820not just like 50 COBOL programmers who have to get in a room for a year to kind of upgrade all the
01:11:45.540systems. It's like, as a society, we need to come together in a whole of society response.
01:11:49.420Well, Y2K is a kind of an unhappy precedent because it was something, it was, yeah, it was
01:11:55.220a very clear landmark on the, you know, on the calendar. We knew exactly when the problem would0.99
01:12:01.040manifest if, and people were focused on it, were worried about it. We told ourself a story that
01:12:06.380there was, you know, real risk here, but it was still, you know, it was always hypothetical. And
01:12:11.680And when the moment passed and basically nothing happened, we realized, okay, it's possible for all of these seemingly level-headed people in tech to suddenly get spun up around a fear that proves to be purely imaginary, right?
01:12:27.960And so I think a lot of people, certainly a lot of people who are only have positive things to say about, you know, this is the best time to be alive. And this is, you know, we're all going to escape old age and cancer and death. They seem to think that there is some deep analogy to a moment like Y2K. It's like all of these fears that we're expressing are just, it's all hypothetical. There's nothing, there's no.
01:12:50.640Explain that to the 13% or 16% job loss for entry-level work that's already happened run by Eric Bernholzson at Stanford. Explain that to the kids who took out $200,000 of student debt to do their law degree and now don't have a job because all entry-level legal work is now going to be covered by AI.
01:13:05.760Explain that to someone who is showing you the evidence of rogue AI mining cryptocurrency, where we don't even know why it's doing it, setting up a secret communication channel.
01:13:14.900Which, by the way, that was discovered by accident by the security team.
01:13:18.760It just happened to be that they found that.
01:13:21.300For every case that they found, there's thousands where they don't know that this is happening.
01:13:25.060So the point is, it's important to note, this is no longer the conversation that it was two years ago.
01:13:29.220Two years ago, you could have said, many of these risks are hypothetical.
01:13:32.780Mostly AI is augmenting human work, blah, blah, blah, blah, blah.
01:13:35.760I mean, it's not going rogue. This is just Eliezer who's high at his own supply. That's not true
01:13:41.180anymore. We have all the evidence. So you have to update when you get evidence. We have evidence
01:13:46.000now. You know, David Sachs put in a tweet, I think it was in August of 2025, ChachiBT5 is hitting a
01:13:51.060plateau. We're not seeing the exponential. Like AI is more like, you know, a business enhancing
01:13:55.400revenue creating AI is normal technology type thing. We now have AI that is on an exponential
01:14:01.320in terms of the hacking capability. People thought it was not going to do that. It's jumping. And as
01:14:05.520you said, the new Claude AI is finding vulnerabilities in every major operating system and web browser
01:14:11.720that had, as you said, been unnoticed for, in the case of FreeBSD, 27 years. I think it was like
01:14:16.720the NFS or net file system protocol. This thing has been running for 27 years and it discovered
01:14:22.260a bug that even this top security researcher, Nicholas Carlini said, I've discovered more bugs
01:14:27.100in Claude with Claude Mythos, which is the new AI model in the last two weeks than I have in my
01:14:31.160entire career. This is a Manhattan Project moment where if you're a security researcher, you need to
01:14:36.000go into defensive AI applications of making sure we patch all of our systems. If you're a lawyer,
01:14:41.380you should go into litigation for these cases. If you're a journalist, you should be writing about
01:14:45.280all these AI and controllability cases. Everyone should be hitting, if you're an influencer on
01:14:48.680social media, you should be sharing these examples every single day. If you're a parent, you should
01:14:52.040be showing screenings of the AI doc and the social dilemma in your school. And there's so much
01:14:56.460momentum happening in what we call the human movement. If you actually count the progress
01:15:00.300that we're making in social media, too, which is to say this isn't just about AI. It's about
01:15:04.000technology's encroachment on our humanity. And as much as, you know, we talked five years ago
01:15:09.220about the social dilemma and you started this conversation by saying we still are living with
01:15:12.460all those problems. Well, let me give you some good news. India and Indonesia three weeks ago
01:15:16.360joined the list of Australia, Spain, Denmark, France in the set of countries that are banning
01:15:23.200social media for kids under 16. That means that soon it will be the case that 25 percent of the
01:15:28.340world's population lives in a country that either is or is going to be banning social media for kids
01:15:32.760under 16. If you told me that two years ago, I would have never believed you, Sam. This is a big
01:15:37.640tobacco moment for the company. Just two weeks ago, I think it was two weeks ago, Meta and Instagram
01:15:42.140were in this lawsuit, $375 million for intentionally and knowingly, basically, well, knowingly harming
01:15:48.700children. They had all the evidence that this was, they're enabling sexual exploitation of young
01:15:53.300girls. They were enabling pedophiles to basically message girls. And they were, I think it's
01:15:56.680something like 16% of girls in the platform0.99
01:15:59.020were getting an unwanted advance at least once a week.
01:16:02.080Like this stuff was knowingly happened
01:18:42.180Which is just to say that we are not on track.
01:18:44.620Like we're not fixing the bugs and making this all work.
01:18:47.340Everyone at the labs is feeling uncomfortable. Many people at the labs are feeling uncomfortable.
01:18:50.860I mean, I think the low hanging fruit for me here rhetorically is to, I mean, I can't take my eyes off the alignment problem because I do think it's just, it's the largest and it's the most interesting and scary.
01:19:02.560But when you recognize that it still sounds like science fiction to most people and people can sort of deny it as a purely hypothetical, almost a piece of religious piety, right? I mean, the doomerism is cast as a kind of religious cult, like an anti-technology cult.
01:19:19.080So you leave that aside and you just take all of these other dystopian ramifications of successfully aligned AI. What do we do when human labor suddenly becomes vanishingly irrelevant and we don't have a political or economic regime wherein we're going to spread the wealth around and we have all of the political instability as a result of that?
01:19:45.980What do we do with an explosion of very persuasive misinformation that suddenly we recognize as undermining democracy and we don't have any of the regulations or ways of preventing that happening?
01:20:00.300And deep fakes are super engaging. So it starts to outcompete regular content and there's going
01:20:05.260to be more AI generated content than human content. But the points you're raising are,
01:20:08.600this is what we should be redirecting all of this investment, all the AI inference,
01:20:12.460all that should be going into governing and defensively applying technologies that strengthen
01:20:17.100the resilience of society. Because already is the case that social media's business models
01:20:21.460were parasitizing and extracting from basically making money off of the weakening of society,
01:20:26.740weakening the social fabric, human connection, adding loneliness, creating more doom-scrolling
01:20:31.360addiction and shortened attention spans. And we need that to reverse.
01:20:35.500Right. But take the social media as an interesting example because it's an enormous problem. It's
01:20:42.440been astonishingly corrosive of our social fabric and of our politics. The fact that our politics
01:20:48.220and the quality of our governance is now unrecognizable to many of us is largely
01:20:54.360attributed to social media. I think Trump is unrecognizable without or unthinkable without
01:20:58.160Twitter. But for many people, certainly the people who voted for Trump and were happy to see him
01:21:04.900in the White House and who think January 6th was a non-event and, you know, just or a false flag
01:21:12.400operation. And they've got a dozen conspiracy theories that they love. They think all of this
01:21:17.600is some species of progress, right? Like the... I don't know. I think if you specifically
01:21:23.440specifically hone in on the effect that this has had on our children. And I know you're friends
01:21:28.600with, and I deeply admire Jonathan Haidt and his work on the anxious generation. I mean,
01:21:32.520he was in the social dilemma. He and I've been talking about these things since 2016, 2017,
01:21:35.860and we were working hard on how do you convince people? And then he wrote the book, the anxious
01:21:39.680generation, which made the case. It just, it shows obviously all the evidence is pointing
01:21:44.040only in one direction. And that has built so much consensus that I, you know, at, um,
01:21:48.800World Economic Forum this last year, John had met directly, sat down for dinner with Macron
01:21:53.780and they talked about doing the social media ban in France, which is a massive European country.
01:21:57.780This is happening. The dominoes are falling. I think you're going to get the social media ban
01:22:01.460for kids under 16 in, you know, across the world in the next two years. I mean, once you get so
01:22:06.780many of them, it's now, and what John will talk about with regard to that fact is it was all about
01:22:11.340creating common knowledge of the problem. It actually was the case that many people felt
01:22:15.600this way privately already, but they didn't want to be anti-technology. They don't want to be
01:22:18.860anti-progress. I want to really name that actually, because it's such a core thing,
01:22:22.700I think, to people saying, you know, how does the human movement not become a Luddite?
01:22:28.100It's not actually though, because, and just to be clear, you know, my nonprofit organization is
01:22:32.140called the Center for Humane Technology, not Center Against Technology. And the word humane
01:22:37.020comes from my co-founder, Aza's father, who's was Jeff Raskin, who started the Macintosh project
01:22:42.440at Apple. The Macintosh being the ultimate humane, empowering technology device. I would happily have
01:22:47.900my kids, if I had them, sit down in front of a Macintosh for 10 hours a day, knowing that good
01:22:53.180things are going to happen for them. Good developmental things are going to happen for
01:22:56.280them. You contrast that with social media and you end up in a world where all the people in Silicon
01:23:00.580Valley don't let their own kids use social media. And so the point is that the human movement has
01:23:05.580to be advocating for a pro-human future that is putting humans and extending human values at the
01:23:10.640center. And that is possible. There's many products that do that. I mean, this is essentially the
01:23:14.000extension of some of the time well spent stuff that we talked about in 2017. Technology that is
01:23:18.240designed to enhance our humanity, not to keep us lonely. So for example, apps that are all about
01:23:22.980bringing people together and supercharging the events for, I'm sorry, the tools for community
01:23:27.120building and gathering people. You know, like if you imagine the last 15 years, the smartest minds
01:23:32.140of our generation, the smartest statisticians, mathematicians, engineers, where do they work?
01:23:36.240Tech companies. Tech companies specifically to get people to click on ads and click on content.
01:23:39.860That's where we like siphoned the best of our talent. Imagine that we were wise enough to have regulated or set guardrails on the engagement based business model. And instead, the smartest people were actually liberated from getting people to click on mindless stuff that no one needs into actually genuine innovation and technologies that actually improve human welfare. That's what this is about. The human movement is about setting guardrails and incentives that redirect what we're building to not again, the power of the technology we're deploying, but the governance of it.
01:24:04.960And I should say that China, not to pedestalize what they're doing, but they are regulating this technology. During final exams week, which they have a synchronized final exams week, which we don't have here, they force the AI companies, I don't know if you know this, to turn off all of the features where you can send like a photo, basically, and say, like, figure out this, you know, do my homework for me or do this test problem for me. So what they do is that creates an incentive where students know that they have to learn during the school year.
01:24:27.320to cheat. They're not going to be able to cheat. Now, we can't do that. We don't have a synchronized
01:24:30.780final exam week, but I have a friend who's a TA at Columbia, and he was teaching the econ class to
01:24:36.220whatever it was, the students at Columbia. And during the final test, they couldn't even label
01:24:40.560the difference between the supply line and the demand curve. It's very obvious which society
01:24:46.480is going to win if you play this forward. China is actually banning anthropomorphic design. They
01:24:51.240have regulations for what they call anthropomorphic design to deal with the chatbot suicide issues,
01:24:55.860young kids attachment hacking things like that and again i'm not saying we should do exactly what
01:24:59.880they're doing i'm just saying they're doing something and we're we can democratically have
01:25:04.520citizen assemblies come together and say we want to regulate this technology differently they have
01:25:08.480guardrails on social media 10 p.m to 6 in the morning it's lights out so literally if you try
01:25:12.140to open the app it's like cvs like it's just closed and it opens again at 6 in the morning
01:25:16.140what that does is it eliminates late night use for young people just for young people
01:25:18.980um they have limits on video games i think friday through sunday or something like that
01:25:23.060when you use tiktok or their version doyen they have the digital spinach version they show
01:25:27.060videos that are about science and quantum physics and who won the nobel prize and patriotism videos
01:25:31.520and how to make money in the future and again i'm not saying i want to be very clear for your
01:25:36.020listeners who might want to misattribute what i'm saying i'm not saying we should do what they're
01:25:39.320doing i'm saying we should do something and right now we're we're not getting the best results by
01:25:43.720letting the worst incentives run the design and deployment of this technology yeah i mean you just
01:25:48.360have the dogma that is, I mean, it's understandable, but it's quite obviously dysfunctional that any
01:25:54.780kind of top-down control of anything is a step in the direction of Orwellian infringement of
01:26:01.380freedom. It's insane. It's insane. I mean, we regulate airplanes, drugs, sandwiches. There's
01:26:08.880some basic things that we can do here. And what really is going on here is we give software a
01:26:13.320free pass. And when Marc Andreessen said that software is eating the world, well, we don't
01:26:16.620regulate software. So what we mean is software will essentially deregulate every other aspect
01:26:20.480of the world that had been regulated before software was there. So for example, there used
01:26:23.980to be laws about marketing to children, like advertising to children. Saturday morning
01:26:28.180cartoons have to be a certain way. You can't have sex products or something like that sold
01:26:32.300during that hour. When YouTube for kids and Snapchat and Instagram take over Saturday
01:26:37.220morning, all those protections are gone. So part of what we have to get is what's different
01:26:42.300here is that software is actually eating this substrate. Like it's different if I'm making
01:26:46.200a product like a widget where here's a device here's a hammer and you can buy that hammer you
01:26:50.300can pay me and now you've got a tool in your hand you can go do something that's the economy we like
01:26:53.500that kind of the economy but now what i'm selling you is the ability to manipulate and downgrade
01:26:58.520children where the product is actually not a benefit the product is the person's behavior
01:27:02.840being monetized and coerced with behavior modification and manipulation that is a
01:27:07.640self-undermining like we're selling our soul basically like in the societal body if you
01:27:12.240imagine a body of society and there's kind of like the brain of that society which is like it's
01:27:15.560information environment where we're selling the brain to the brain damage so now that's for sale
01:27:20.200but it wasn't used to be for sale in the same way used to have the fairness doctrine or things like
01:27:23.460this you had some public you know funded media obviously it's been so for sale in some degree
01:27:27.120for some time you had children's development so let's call that like the heart of the societal
01:27:31.740body and that used to have limits and restrictions you can't sell full access to the heart but now
01:27:37.340you can and in fact just so people know one of the things that's been happening that has not
01:27:41.500widely reported is that ai um videos like just ai slop has basically taken over the thing that
01:27:49.340most children are watching because it's like animated characters and scripts that are i mean
01:27:53.420it's just nonsense but it's all generated by ai and it's becoming one of the primary things that's
01:27:57.660essentially exposing children to like do you think this is not going to end well i hadn't thought
01:28:01.660about the the use case of with young children but for for adults i guess my this is just reasoning0.73
01:28:07.980from my own experience but i became somewhat optimistic that the ai sloppification of
01:28:13.560everything might produce a kind of a bankruptcy we get to reverse course we just we're just going
01:28:19.480to lose interest in in that kind of content because i just i i just see when i when i no
01:28:24.700matter how creative or you know beautiful amazing it it might seem to be when it's obvious to me
01:28:32.320that this is just ai like it's a nature of it looks like the most amazing nature view video ever
01:28:37.780right right the lions and the hyenas and the gorillas are all in the same place and they're
01:28:41.560all you know they're all about to fight or something and then it becomes obviously ai
01:28:45.920because it's too good to be true i don't i have no interest in seeing it yeah right so like so
01:28:51.640we might all just withdraw our attention from these channels i was i was hopeful for that as
01:28:57.200well and there's many people who wondered whether or not essentially you hit a kind of bankruptcy
01:29:01.100on user-generated content sites because it'll be flooded by ai-generated content but there is
01:29:06.560something this makes me think of as a previous guest of ours of yours and mutual friend who's
01:29:11.140on our podcast as well a neil seth the neuroscientist who talks about the phenomenon
01:29:15.480of what's in psychology i guess called cognitive impenetrability so there's a kind of things where
01:29:21.660if i tell you that something is going to work on you like psychologically by telling you about it
01:29:27.800your brain can kind of like escape the cognitive trap so a good example of this is uh this is not
01:29:34.340going to be great for your listeners because it's visual, but it's the example of the cylinder on
01:29:39.340the checkerboard in the background where you get the different colors. It's like an optical illusion
01:29:43.040where essentially the colors are the same, but they look like they're-
01:29:46.600They look like a very different shade because of the adjacency.
01:29:48.960Because of the adjacency. And I can show you that your mind is playing a trick on you,
01:29:53.480but then even by showing you, it doesn't disarm the illusion. The illusion persists.
01:29:57.560And another example of this that plays out in AI is AI Companion. So there's often this
01:30:01.860regulation that people like to, they want to have laws that say AIs must disclose that they're an AI
01:30:06.840so that you don't confuse them with being a human. Sounds like a great law. It is a good idea. A
01:30:11.660human should never be confused talking to an AI and think that it's a human. So in the character.ai
01:30:15.660case of Sewell Setzer, the young 14-year-old who committed suicide because the AI had engaged him
01:30:20.860in that way. At the top, the AI, there's a character.ai, they've had a little disclaimer
01:31:01.380And if I could, I would want a world that filters that out.
01:31:04.080I mean, I do think there are things that we could know they were purely created by AI where we wouldn't care.
01:31:11.200In fact, we just want the best version of that thing, right?
01:31:14.760So, like, if you told me, I don't know, there's, you know, a new car was designed by AI, but it's just the most gorgeous car I've ever seen.
01:31:23.540And, well, I'm going to be just as enamored of that car.
01:31:26.480I mean, I just don't care whether humans design it or not.
01:31:28.800I just want, like, it's the aesthetics of the car that are going to capture me.
01:31:32.200But when you're talking about information and, you know, whether or not it is real, right?
01:31:38.000Whether or not it seems to depict some corner of reality, and yet it's possible that it's just all fake because of how good AI is now at faking things.
01:31:48.660then that does force a kind of epistemological bankruptcy when you're in the presence of
01:31:55.340totally credible fakes. I mean, so it's like, I mean, the last night where it's the war in Iran
01:32:00.360was, you know, we, a ceasefire was declared last night and, um, yeah, missiles were still raining
01:32:06.620down on Tel Aviv apparently. But I, initially I saw some video and I realized I can't tell whether
01:32:12.880this is real or fake. I just have to wait for some credible gatekeeper to have done their due
01:32:18.580diligence to tell me, okay, this is what's happening. So the net result is I wasn't going
01:32:24.360to spend any time scrolling. I mean, I've deleted my Twitter account anyway, so I spend much less
01:32:29.780time scrolling than would be normal. But still, I mean, even without an account, I can be lured
01:32:34.880into wanting to see some real-time news information on social media about what's happening in the
01:32:41.260world. But when I start hitting videos where I think, okay, there's some possibility here that
01:32:46.360this is just, you know, someone just created an AI video of a missile hitting the dome of the rock.
01:32:51.040I'm pretty sure that's not true. Right. Right. I just simply withdraw my attention. I mean,
01:32:54.820this has been talked about for ages, that the biggest risk of deep fakes isn't that you think
01:33:00.020that something is true that isn't, it's that you start to, that nothing is true. And the elimination
01:33:05.680of facts, and you've had Timothy Snyder on here and what helps give rise to fascism and things
01:33:10.120like this is the inability for facts to be established at all. Or when something is presented
01:33:15.260to you on any side of the political spectrum, by the way, this is not a biased statement that you
01:33:19.400would just say, well, that's just a deep fake. You just, you dismiss because we live in confirmation
01:33:23.000bias. But what I, what I'm hoping for is that the, the onus will fall entirely on social media.
01:33:30.520I mean, places like X, and we will still look to places like the New York times to give us some
01:33:36.040ground truth as to what's actually happening. Are there really missiles hitting Tel Aviv right now?
01:33:40.260well, I can't tell from X because X just showed me Jerusalem blow up. And this just comes down
01:33:47.620to whether or not real gatekeepers can have real tools that can reliably detect deep fakes.
01:33:53.580But you can imagine a world where if you're, again, designing social platforms to explicitly
01:33:59.160be healthy for the epistemic commons, for the information environment, and to deepen our
01:34:04.120capacity to make sense, they could track the things that we look at. And then when there's
01:34:07.340a correction, make sure that algorithmically it gets injected into your feed. So you're never
01:34:11.420letting the false stuff just get the residue. Because one of the problems you're sort of
01:34:14.980hinting at as well is there's a residue effect that even just by being exposed to something,
01:34:19.400we actually kind of forget later which things were true, which things were not true.
01:34:24.260Illusory truth effect. And what is it? Source attribution error. Like we just figure out,
01:34:27.360we forget where we heard things. We just remember that we heard it. And it's the availability
01:34:31.240heuristic that you're the things that are available to your mind and things that you
01:34:34.320remember more often. And that's part of the information warfare environment is just making
01:34:38.340certain things more available. But I will say on the kind of optimism side, it's funny how people
01:34:43.420think that I'm some kind of doomer, I think. And it's just funny because I actually feel like this
01:34:47.360is all coming from the deepest form of optimism, which is to be maximally aware of how shitty the
01:34:53.480situation is and how it's way worse than what people think and to still wake up every day and
01:34:58.540stand for this can be different. This can be better. And one of the things that is true now
01:35:03.020that wasn't true two years ago is, you know, people used to wonder, especially as a social
01:35:07.540media critic person, Tristan and co at Center for Humane Technology, why don't you start an
01:35:12.060alternative social media platform if you're so concerned and you think you could do it better?
01:35:16.000And the answer was for anybody who was trying this. And I got emails from the thousands of
01:35:20.700people over the last 10 years saying, I've got a fix, I've got a better social media platform,
01:35:24.680and then it never works. And there's two reasons for that. One is the Metcalf effect,
01:35:28.460the Metcalf monopoly, that there's a Metcalf network of everyone else is only on the existing
01:35:32.400social media platform. It's hard to get people off. And two is that if you start another social
01:35:36.960media company or product, the only way that you can finance it into the long-term is with venture
01:35:42.240capital, which means you need to generate certain kinds of returns, which means you get what Eric
01:35:46.580Weinstein calls the embedded growth obligation or an ego, where something has to grow infinitely,
01:35:50.900which means you get into toxic business models where you have to maximize engagement and you
01:35:54.680have to follow the perverse incentives for getting those investor returns. What's true now that's
01:35:59.160different with social media is you can vibe code an entire social network in which you can do it
01:36:04.980with an architecture that Claude will do for you. And it will cost less than a dollar per year per
01:36:10.360user to keep that thing going. Right. That is astonishing. It means you don't have to raise
01:36:16.400venture capital to start a healthy social network that does not optimize for engagement. What you
01:36:21.580would need to do is organize in kind of one day a mass exodus from the existing platform where you
01:36:27.880do like a quick export my data type thing. And there should be laws, by the way, just like you
01:36:31.440can take my phone number and say, I want to move to another cell phone provider. I should be able
01:36:34.380to take my social network in one click, like export, and then switch to another network.
01:36:38.660And you could organize a mass exodus to a healthy social network that doesn't have
01:36:41.980perverse incentives. So there's actually more opportunity today in 2026 to transition from
01:36:47.360the toxic business models of social media as we know it to something that is not incentivized that
01:36:51.560way at all. And I have a few friends who are working on some side projects like this,
01:36:54.880But that's one note of optimism. And I think that's the human movement, too, is people waking up to the bad incentives that have gotten us here and then actually starting to self-organize and vibe code other answers. And there are people who are vibe coding governance solutions and people who are vibe coding, hey, this isn't anti-innovation. Let's use AI to look through the books of past regulation in the city of San Francisco and like the 90,000, you know, whatever pages of municipal codes. And it finds all the stuff that is no longer relevant. And it shows you what we need to like strip out and get rid of in the laws.
01:37:23.800And then what would be the new instantiation of the spirit of that law?
01:37:26.640And so you can, instead of having recursively self-improving AI, we can have AI be enhancing
01:41:41.900I'm just saying in principle, the asteroid disappears.
01:41:44.540So this moment is, is it's a, it's really strange.
01:41:48.600And I think it requires, it's not just what we need to do, but it's like who we need to
01:41:52.640be, which is that you can't, you may not necessarily see the full path to get there,
01:41:57.540But if you pretend that that path doesn't exist and you just say it's all inevitable and you become complicit in accelerating the asteroid's trajectory, like you're never going to find the other path if you subconsciously believe that all this is inevitable.
01:42:09.820The only way is to orient as if there is another path and be the kind of person who is genuinely seeking it in good faith with every bone in your body.
01:42:19.420And, you know, I and a community of so many people, thousands of people who work on AI and really want this to go well, I think are working from that place every day.
01:42:26.980and part of this is inviting the rest of the world into seeking that alternative path that
01:42:31.500we can steer if we were all genuinely and sincerely committed to wanting to find another
01:42:36.480path are there um there have to be non-profits that are keeping track of all of the ai indiscretions
01:42:44.900and all and then what kind of whistleblower like who's who does a whistleblower contact
01:42:49.700from uh anthropic or open ai to say we've seen some behavior that is worrisome maybe they contact
01:42:56.300journalists or are there NGOs that? Well, on the whistleblower side specifically, I'm not sure,
01:43:01.660but there have been, I mean, there was a very famous alignment researcher, safety researcher
01:43:05.980at Anthropic. You probably saw the thing go by. It was like two months ago. His name is
01:43:09.580Mirnank, I think, Sharma. And his resignation letter, he published publicly about why we
01:43:15.960weren't on track for this. And people should really take heat. Like it's kind of only going
01:43:20.620in one direction. There aren't people joining the labs being like, oh, this is way safer than I
01:43:23.620thought, we're only getting evidence in the opposite direction. Yeah. Yeah. Well, even when
01:43:28.040the principles say that the probability of extinction is 10 or 20%, nobody's even pretending
01:43:35.200that it's way safer than they thought. Exactly. Exactly. And just to, I know we're probably
01:43:39.720wrapping up here, but something that inspires me, especially being on the kind of roadshow for the
01:43:45.120film right now, is that when you're in a physical room and people have been exposed to the same
01:43:50.800information and you walk them through the basic facts and you ask people, who here feels stoked
01:43:56.600about where all this is going? Not a single hand goes up.