The Megyn Kelly Show - The Tucker Carlson Show： AI Whistleblower： OpenAI Scandal, AI Cults, Neuralink & Our Last Chance to Stop the Tech Oligarchs

00:00:00 Hey, Megyn Kelly Show listeners, it's Tucker Carlson.

00:00:02 There has been a lot of speculation and panic

00:00:05 about what the age of AI means for humanity.

00:00:08 Are we creating technology that will become uncontrollable?

00:00:12 Experts who are building AI said that's a real possibility.

00:00:16 We recently sat down with someone

00:00:17 who understands what is happening.

00:00:19 His name is Nate Suarez.

00:00:21 He's a computer scientist who worked at Google

00:00:23 and the Defense Department.

00:00:25 So when he says AI is on a path

00:00:27 to killing every human being on Earth,

00:00:30 It's worth listening. 0.98

00:00:31 This is one of the most,

00:00:32 maybe the most important conversations

00:00:34 taking place in the world today.

00:00:36 If you want to listen to conversations like this

00:00:39 about things that actually matter,

00:00:40 learn what's happening,

00:00:41 what your future may look like,

00:00:42 we hope you'll check out our show.

00:00:44 New episodes of The Tucker Carlson Show

00:00:46 are released every Monday, Wednesday, and Friday.

00:00:49 Follow the show right here in your podcast feed

00:00:51 so you don't miss a single one.

00:00:57 Nate, thank you so much for doing this.

00:01:00 Um, you've written the world's darkest book. You've devoted your life to warning the world

00:01:06 about the potential dangers of AI. So let's just start by hearing your explanation of why

00:01:13 AI is dangerous in addition to just being annoying. Yeah. You know, the, the, the very basic

00:01:20 common sense point is if you race to make machines that are much smarter than any human,

00:01:30 to make machines that are more capable

00:01:32 at inventing their own technology than humans are.

00:01:37 And you race into this without really knowing what you're doing.

00:01:40 The most likely outcome is just that the machines get loose

00:01:45 and do their own thing,

00:01:46 and that humanity dies as a side effect,

00:01:49 just like humanity has killed lots of other animal species,

00:01:53 not because we hate them, but just as a side effect.

00:01:54 uh and in in some sense a lot of people find it sort of obvious or intuitive that if you just

00:02:03 like make these really smart really powerful machines why would they care about us uh

00:02:08 and i think that intuition basically is right and there's a lot of arguments you can have on

00:02:14 each side you can get in the technical details uh but my book is basically just um

00:02:20 getting into those details

00:02:23 and saying like,

00:02:23 yep, it sort of holds up.

00:02:25 If we make smarter machines

00:02:26 without knowing what we're doing,

00:02:27 it's just going to go poorly.

00:02:32 Are we actually going to make machines

00:02:34 capable of what you're describing?

00:02:37 You know, the companies are trying to.

00:02:40 They talk about how they are pursuing

00:02:44 superintelligence in the true sense of the word.

00:02:46 That's Sam Altman's phrase.

00:02:48 Dario Modi of Anthropics 1.00

00:02:49 as they're trying to make the equivalent

00:02:51 of a country worth of geniuses in a data center.

00:02:54 So these guys are really,

00:02:55 you know, these aren't chatbot companies.

00:02:58 They didn't set out to be chatbot companies.

00:03:01 They set out to make these machines

00:03:03 that can sort of exceed humans in every way.

00:03:06 And that's what they're targeting.

00:03:07 There's a separate question

00:03:08 of whether they will get there.

00:03:09 Why would anyone want to build a machine

00:03:12 smarter than people?

00:03:14 I think that a lot of them

00:03:19 hope that they'll be able to make the world much, much better. You know, they hope for

00:03:23 a cure to cancer and then more than a cure to cancer, they hope for a cure to aging.

00:03:28 They hope for, you know, a thousand years worth of technological development compressed into two

00:03:35 years. And I sort of don't think that they're going to be able to get that really or harness

00:03:46 that for good ends,

00:03:48 but that's what I think

00:03:49 they're shooting for.

00:03:52 So they're not,

00:03:52 but I mean,

00:03:53 the core point you're making

00:03:54 is these companies

00:03:55 did not set out

00:03:56 to make consumer products,

00:03:58 like to make your life better

00:03:59 necessarily in the short term

00:04:01 with a more efficient

00:04:02 search engine.

00:04:04 That's right.

00:04:04 Yeah, you know,

00:04:05 OpenAI started before

00:04:07 the chatbots,

00:04:10 the large language models

00:04:10 were even a thing.

00:04:12 They were started,

00:04:13 forget those late 15

00:04:14 or early 2016,

00:04:16 But the paper that unlocked the most recent wave of AI came out in 2017, which was after OpenAI was founded.

00:04:25 And these guys are, the large language models are a surprise revenue stream.

00:04:35 And that revenue stream can fund the creation of even more, even larger data centers for the next level of the technology.

00:04:43 But these guys have their eye on the sort of ultimate version of the technology, which is these machines much smarter than humans.

00:04:51 And that's sort of the ultimate version, because once the AIs are smarter than the humans, the AIs can carry on the AI research and make the next generation of the AIs, which make the next generation of the AIs.

00:05:01 And that's sort of what they're shooting for.

00:05:04 What is superintelligence?

00:05:06 we in our book define super intelligence as an ai that is uh better than the best humans

00:05:14 at every mental task so uh anything that a human can do purely mentally the ai can do that better

00:05:21 one one thing people often get caught up on about this is uh that includes the ai being better at

00:05:28 things like persuading humans at things like charisma you know we often think of intelligence

00:05:33 as the stuff that the nerds have

00:05:35 and that the jocks like.

00:05:37 You win the chess match.

00:05:39 Right, it's the chess guys

00:05:40 rather than the politicians guys

00:05:42 or the rock stars.

00:05:44 But that's not really what

00:05:46 the intelligence in artificial intelligence means.

00:05:50 The intelligence in artificial intelligence

00:05:51 is about the stuff that humans have

00:05:54 and that mice don't.

00:05:56 It's sort of the whole package.

00:05:59 You know, like the politician

00:06:00 who's very charismatic

00:06:03 It's not that like chess playing happens in your brain

00:06:06 and charisma happens in your kidneys, right?

00:06:09 They're sort of both mental functions.

00:06:11 Yes.

00:06:12 And so the sort of super intelligent AIs

00:06:17 are not just super good at playing chess.

00:06:20 They're also good at super persuasion

00:06:21 and they're super good at the research

00:06:24 and at the technological invention.

00:06:27 So super intelligence as you define it

00:06:29 is a machine that is better at every mental function than any human being.

00:06:36 That's right.

00:06:37 And, you know, the definition, that doesn't mean that nothing crazy will happen on this 0.73

00:06:45 planet until the AIs are super intelligent.

00:06:48 Super intelligence in this definition is sort of a point where past this point, things must

00:06:52 be pretty crazy.

00:06:54 Because now the AIs can do automated AI research and can make smarter AIs and they can figure

00:06:58 out how to run the robot factories and they can build the better robots and all that.

00:07:02 You could have things start to get crazy before you have a super intelligence in this sense.

00:07:07 You could have AIs that are much better than humans at some tasks and much worse than others

00:07:11 that are still causing all sorts of crazy happenstances. The super intelligence is sort

00:07:19 of a like, once you get here, stuff's crazy point. It's not a things will stay normal until you get

00:07:25 here point. So why is superintelligence the significant milestone that you're worried about?

00:07:33 I mean, I would say that I'm also worried about what will happen before that milestone. It's more

00:07:41 like, like having the definition makes it easy to talk about like, how crazy would things get

00:07:47 once you are past this point or sort of like, you know, let's like, if we assume that the

00:07:54 machines get here, what happens? And then the answer is, it would be pretty crazy. It would

00:07:59 be pretty wild. And it sort of lets you factor the conversation into, how to say,

00:08:11 I think in the artificial intelligence conversation, there's actually a lot of

00:08:14 conversations going on. One conversation is like, can the machines really get smarter than the

00:08:18 humans? One conversation is like, how fast can we get there? Is the current technology on that

00:08:24 route. Another conversation is what happens if we do get there? Like, you know, would the AIs

00:08:31 care about us? Would they not care about us? What would they be able to, like, what would they try

00:08:35 to do? There's another question, which is what would they be able to do, right? And these are

00:08:39 all sort of different conversations about AI. And the superintelligence definition, it's not really

00:08:44 like here's where all the fears hang. It's more like it lets us break those pieces out separately

00:08:51 and be like, well, is superintelligence possible

00:08:54 separate from, well, what would happen if you had one?

00:08:57 What would happen if you had one?

00:08:59 Most likely outcome, I think destruction of the planet.

00:09:03 So our friends at Preborn, who are actual friends, by the way,

00:09:05 just asked us to thank the listeners of this show

00:09:08 for supporting the pro-life cause.

00:09:09 A lot of people do all of a sudden, trust me,

00:09:14 because what's happening is crazy

00:09:16 and people are waking up to that.

00:09:18 The president of the group is our friend, Dan Steiner,

00:09:20 and he wants Americans to know how much of a difference they have made.

00:09:25 This year's donations have helped save 50,000 babies from abortion.

00:09:29 That's enough to fill a professional baseball stadium.

00:09:31 50,000 lives saved, as well as generations that will follow them.

00:09:35 Pre-born wastes zero money.

00:09:38 We're not big on charities because a lot of them waste money.

00:09:40 Pre-born does not.

00:09:42 He uses every tax-deductible gift to rescue babies

00:09:44 and share the hope of Jesus with their moms.

00:09:47 In 2026 alone, over 8,000 women made the decision to follow Christ.

00:09:52 The generous support of Preborn's donors made that all possible.

00:09:56 $28 to provide one life-saving ultrasound,

00:10:00 just giving expectant mothers the knowledge that they have a baby that they're carrying. 0.87

00:10:05 $15,000 could place an ultrasound machine in a pregnancy clinic where it can save countless lives.

00:10:10 Dan Steiner's message is really simple.

00:10:12 Thank you for helping.

00:10:13 Your generosity is literally saving lives.

00:10:15 To help Preborn save more, dial pound 250 and say the keyword baby.

00:10:20 That's pound 250 baby.

00:10:22 Or visit preborn.com slash Tucker.

00:10:24 preborn.com slash Tucker.

00:10:27 What would that look like?

00:10:30 So I'm going to be a little bit annoying here and say a couple caveats first.

00:10:37 Because it's sort of a tricky one to sort of predict things that are smarter than us.

00:10:43 Yes.

00:10:43 and the the first annoying caveat i will give is uh if you go to play a game of chess against

00:10:52 magnus carlson i predict you will lose no offense uh he's just the best human chess player alive

00:11:00 if you then ask what piece will use to checkmate me i'm like well that's a much harder question

00:11:10 Yes.

00:11:10 Right? It sort of is like very easy to predict the winner. It's hard to predict the exact methods.

00:11:16 So my prediction that humanity ultimately dies is a different sort of prediction than my prediction

00:11:25 about like, it could go this way, it could go that way. So I'll give you some stories,

00:11:30 but these are stories that are like, well, maybe Magnus will like fork your queen and your rook

00:11:36 with his knight and then finally use the queen to checkmate you. Like, yeah, that could happen,

00:11:40 but it's a different sort of prediction. It's a guess, right? The guess here is

00:11:47 that if we manage to make these really smart AIs, they will have their own stuff that they're

00:11:58 pursuing, which is not quite what we wanted, not quite what we asked for. It'll have this

00:12:02 other strange stuff.

00:12:04 There's a whole discussion

00:12:05 about why that happens,

00:12:06 but we're already starting

00:12:07 to see it in practice

00:12:08 with some of the recent events.

00:12:12 And

00:12:12 they would be able

00:12:16 to pursue

00:12:17 whatever it is

00:12:18 they're pursuing

00:12:18 much more efficiently

00:12:21 than humanity can.

00:12:25 And so we're talking about,

00:12:26 you know,

00:12:26 automated factories

00:12:27 that produce the robots,

00:12:29 that produce the factories

00:12:30 in a fully automated

00:12:31 supply chain.

00:12:32 And then if the AIs have anything they're trying to do that they can do more of with more resources, they start gobbling up the resources on the planet.

00:12:43 Sort of like how humanity spread and started gobbling up all the resources on the planet.

00:12:48 And the sort of most basic thing to visualize here might be you have factories that produce robots that produce factories that produce robots that also build data centers.

00:12:59 and you just have these, you know,

00:13:01 fully autonomous, self-replicating ecosystems

00:13:05 of robots and factories and data centers

00:13:07 that don't care about the instructions

00:13:08 humans gave them.

00:13:11 And that cover the planet,

00:13:14 take all the resources,

00:13:15 take all the sunlight,

00:13:16 take all the places we were growing crops,

00:13:18 probably raise the temperature of the planet

00:13:20 because you can compute more efficiently.

00:13:23 Well, technically the earth radiates more heat

00:13:25 when the world is hotter

00:13:25 and that lets you sort of do more computation.

00:13:29 And then sort of, you know...

00:13:30 So it's good for the machines to have a hot planet.

00:13:32 It's good for the machines to have a hot planet.

00:13:34 Yeah, the sort of physical limits

00:13:36 on how much computing you can run on the surface of Earth

00:13:39 is bounded by how much heat you can radiate to space.

00:13:42 That's sort of the first constraint you bump up against.

00:13:45 You might think that it's energy,

00:13:46 but actually there's a lot of helium, hydrogen to fuse

00:13:48 on this planet.

00:13:49 So you can get plenty of energy.

00:13:51 What you need is heat dissipation.

00:13:54 And the world can dissipate more heat

00:13:56 when it's hot.

00:13:59 So, if you're, you know, imagining some collective of AIs that are trying to run a lot of computing power, trying to do a lot of computation for one reason or another, they sort of prefer the planet running hotter.

00:14:11 And so, it sort of is like nothing personal.

00:14:13 It's just if you let these AIs get out of control, they transform the planet into something unlivable.

00:14:22 How hot?

00:14:23 uh as hot as uh you can while still having the computers not melt so probably hundreds of degrees

00:14:34 um at which point you hear people say well then just turn it off

00:14:40 uh so i think we do have an opportunity to turn it off you know and i'm not here saying that we're

00:14:47 going to die i'm here saying we sort of are going to need to act you got to be careful about the

00:14:52 turning it off piece because your opportunities to turn the AI off only last when the AI is running

00:15:01 on the computers you know it's running on. If you have the AIs breaking out and running on hidden

00:15:06 computers, if you have the AIs running robot factories that produce robots that are under

00:15:13 the control of the AI, where those robots can then go build more computers that are not hooked

00:15:16 up to your network

00:15:17 that can run the AIs.

00:15:19 These are sort of thresholds

00:15:21 where the AI is able

00:15:22 to keep itself running

00:15:23 and you better have turned it off

00:15:26 before that point.

00:15:30 Because after that point,

00:15:32 it's impossible.

00:15:34 Yeah, I mean,

00:15:36 the,

00:15:38 if,

00:15:41 it also gets a lot harder

00:15:43 if the AI knows

00:15:44 you're going to try to shut it off

00:15:45 and is going to try to stop this.

00:15:46 You got to remember that we're talking about things that are smart here. And so if the AI sort of sees it coming, maybe it defects to North Korea, where it can convince them to run it on its data centers in ways that initially benefit them, but that ultimately benefit the AI.

00:16:02 I think we're going to have to redefine what life is because you're describing a living autonomous thing.

00:16:08 Yeah, artificial life, in a sense.

00:16:11 I mean, in some sense, we're already seeing the very, very beginnings of that.

00:16:17 But yeah, once the AIs can replicate, once they can improve themselves,

00:16:24 once they can find ways to run where you don't know that they're running,

00:16:30 once they can

00:16:33 sort of

00:16:36 run the robot factories

00:16:39 to produce the robots

00:16:40 that can produce more factories

00:16:42 and that can produce computers

00:16:43 that the AIs can run on,

00:16:44 then yeah,

00:16:44 you have in some sense

00:16:45 made a new artificial life form.

00:16:53 And, you know,

00:16:53 humanity is on the top

00:16:54 of the food chain right now

00:16:55 because we

00:16:56 are sort of the only

00:16:58 smart life form around.

00:17:00 yeah if you suddenly make a new one that is smarter that is able to make a million copies

00:17:07 of itself that is able to run faster it's just kind of a crazy thing to race into

00:17:13 it's a weird thing to want and as it's developed i was going to say slowly but it hasn't been

00:17:22 particularly slow 10 years the rest of us have watched as people who i mean put it in political

00:17:31 terms don't share the values of most americans are now in charge of this and it's almost like

00:17:37 everyone sat passively by as it happened yeah i i think i think there's a lot going on there

00:17:44 i think part of it is that most people didn't and some still don't believe that ai was going

00:17:50 to be able to keep going. I think there's a lot of like putting the head in the sand and being

00:17:55 like, oh, it's, you know, just slop. It's going to be a bubble. It's going to pop. It's going to

00:17:59 hit a wall. It's not going to be able to improve anymore. And I think a lot of that was sort of

00:18:06 wishful thinking. And I think that at the very least, if people are like, we are making, you

00:18:12 know, the super intelligent machines that are going to replace humanity as the top of the food

00:18:16 chain because we think it's going to go great i think humanity's response should not be go ahead

00:18:23 and try we hope you'll fail i think the response should be like hold on that's kind of crazy um

00:18:30 yeah i had i had some other point too but i've forgotten it so how uh predictable has the

00:18:38 evolution of ai been has it taken turns that you didn't expect yeah totally uh i was

00:18:46 So back before the large language models.

00:18:50 How long have you been following this issue?

00:18:51 I started following it in 2012 and I started working on, I started working with some of the people trying to make this go well in 2013 and I started full-time in 2014.

00:19:02 A long time.

00:19:03 So over a decade.

00:19:05 What has surprised you?

00:19:06 You know, the large language models, they have gone further than I expected initially.

00:19:15 And I think one of the big surprises here

00:19:18 was AI undergoing a phase

00:19:21 where everybody can see it.

00:19:24 You know, back before the large language models,

00:19:28 really only nerds paid attention to AI.

00:19:32 And it wasn't that there was nothing happening.

00:19:34 You know, we were watching the, you know,

00:19:37 Google DeepMind make an AI

00:19:39 that could beat the best Go player.

00:19:43 Yes.

00:19:43 And that was sort of a milestone for people who were paying attention.

00:19:48 But for all we knew, the labs were going to keep on working on engineering problems and

00:19:55 keep on working on the relatively nerdier problems.

00:19:57 And you were never going to have like a mass market consumable product.

00:20:01 And so it was like, for all we knew, it would just stay in the labs and no one would ever

00:20:04 really notice that these guys were gambling with the whole future.

00:20:08 Now, at least, AI is everywhere.

00:20:11 Everyone's starting to have the conversation.

00:20:12 And that is in some sense actually really quite optimistic because it gives the world an opportunity to see what these guys are trying to do and say, hold on.

00:20:24 I guess you'd flip it around and conclude that because the overwhelming majority of people seem to be very opposed to AI, I don't even know anyone who's for it.

00:20:33 Every college commencement speaker who mentions it gets booed.

00:20:36 Yeah.

00:20:37 and that view that opposition to it has had no effect at all in slowing it down

00:20:44 i don't know does that give you hope um i mean

00:20:50 i i hear this a lot i would say when you're coming at this from 2012

00:20:57 yeah it really feels like we're making progress right no that's fair it doesn't feel like we're

00:21:01 all the way there yet um but can i just make the point china china

00:21:07 China, China, China. We have to do this because of China. Yeah, you know, I think China and the

00:21:14 USA have a shared interest in not dying to a rogue superintelligence. Storm season is on the way,

00:21:21 and that means your Wi-Fi could go down. And if it does, your security cameras go dark too. So

00:21:26 at the exact moment you want to survey your property during the blackout, you can't. Unless

00:21:32 you install cameras by defend defend cameras run on cellular not wi-fi and that means they keep

00:21:37 working even when you lose power you don't choose a carrier you don't sign a contract you just mount

00:21:43 it open the defend app and bam you're connected it's that simple so whether it's your ranch your

00:21:48 job site your backyard defend works where wi-fi doesn't and keeps working when the grid goes down

00:21:54 which it may plan started just five dollars a month most people pay around 10 plus there are

00:21:58 or no binding contracts,

00:22:00 meaning you get to cancel whenever you want.

00:22:02 So unlike big tech,

00:22:03 Defend will never sell your data,

00:22:04 not to China, not to the government,

00:22:05 not to anybody.

00:22:06 It's an American company built for people

00:22:08 who want actual security.

00:22:11 Visit defendcellcam.com

00:22:13 and use the code Tucker for 20% off.

00:22:15 That's defendcellcam.com, code Tucker.

00:22:19 Is it conceivable that, 0.76

00:22:22 like, let's just say China can,

00:22:23 I'm sure, by the way,

00:22:25 that China is showing more strength

00:22:27 than we are in this.

00:22:27 yeah i mean and if nothing else they sort of uh have a lot more reason to censor their ais

00:22:34 right and to try to make them you know not say certain things to the to the population so um

00:22:40 but if the united states you know if the u.s government were somehow able to get control of

00:22:45 the tech sector which is at present not possible but let's say it did

00:22:50 let's say the president was more powerful than the tech oligarchs which he is not but

00:22:56 for the sake of argument.

00:22:59 Let's see, he was and he shut it down. 1.00

00:23:01 Would that matter if some Indian lab 0.99

00:23:04 or Chinese lab created it? 0.98

00:23:06 So it does have to be a global stop

00:23:08 to this creation of superintelligence.

00:23:10 I think there's a couple of reasons

00:23:12 why this is possible.

00:23:13 One is I think a lot of people,

00:23:15 they hear that some parts of AI need to be stopped

00:23:21 and they think I'm saying

00:23:24 that all parts of AI need to be stopped.

00:23:26 And, you know, there's all sorts of interesting issues with, you know, AI and education and AI and, you know, military drones that society has to wrestle with.

00:23:39 But these are all sort of separate issues from the superintelligence race.

00:23:43 And if you sort of like tease those issues apart, it becomes much more possible to do like a more surgical intervention on we're not going to race towards the superintelligence in ways that leave a lot of the rest of the sector alive.

00:23:55 which um which which makes it sort of an easier coordination effort to to attempt

00:24:07 and on that front on the superintelligence front it's not that you know just shutting down the

00:24:15 the domestic race towards superintelligence would be enough but uh

00:24:20 racing towards super intelligence requires a huge amount of highly advanced computer chips

00:24:27 that can only be produced as sort of the peak of the global supply chain yes uh which is largely

00:24:35 controlled by u.s allies and so there is absolutely a possibility that uh a u.s-led

00:24:45 coordination effort. Could say, look, we're not doing the super intelligence thing. We're going

00:24:51 to monitor the extremely heavy chip concentrations of like, you know, 10 or 100,000 of these highly

00:24:57 advanced AI chips. We are going to make sure that they are not doing these super intelligence

00:25:05 training runs. And I think there's a way, if the US was leading that effort, there would be a way

00:25:14 to get China on board

00:25:16 and set this up in a way

00:25:17 that was monitorable,

00:25:18 enforceable, verifiable.

00:25:20 Yes.

00:25:21 We sort of,

00:25:22 in many ways,

00:25:23 it would be easier

00:25:23 than a nuclear arms treaty

00:25:24 because uranium is a rock

00:25:27 you can dig out of the ground,

00:25:28 whereas these highly advanced

00:25:29 computer chips

00:25:30 sort of only come out

00:25:30 of one fab in Taiwan.

00:25:32 You know, and so it's like...

00:25:33 These are TSMC chips.

00:25:34 Yeah.

00:25:36 And so,

00:25:37 and there's other parts

00:25:38 of the supply chain

00:25:38 that are very narrow,

00:25:39 like the lithography machines

00:25:40 that come out of the Netherlands.

00:25:42 And so,

00:25:43 So, yeah, we could absolutely lead a global effort

00:25:49 to say we're not making the superintelligent machines.

00:25:51 And it would be a bit tricky,

00:25:53 but it's just, we could just do it

00:25:54 if we had the political will.

00:25:57 I think that people either don't know what's happening

00:26:02 or they assume that like a lot of fears,

00:26:06 this fear will turn out to be groundless.

00:26:10 People point to Y2K.

00:26:11 yeah um you know i hope the fears are groundless i think with a lot of past fears uh the

00:26:19 what happened is not that the fear was groundless it's that

00:26:24 uh people noticed the issue and put in a ton of work to make the bad thing uh not happen

00:26:32 you know i think we saw this with the hole in the ozone layer where people are like oh whatever

00:26:37 happened to the hole in the ozone layer well what happened with the hole in the ozone layer

00:26:39 is that we fixed it.

00:26:40 It's not that it was fake.

00:26:42 It's that, you know,

00:26:43 we went and banned

00:26:43 the chlorofluorocarbons

00:26:44 that were actually blowing

00:26:45 this hole in the ozone layer.

00:26:46 We found other ways to,

00:26:48 you know, cool refrigerators

00:26:49 that worked similarly well

00:26:52 and didn't, you know,

00:26:54 put this hole in the ozone layer. 0.95

00:26:56 I think Y2K was actually 0.99

00:26:57 one of these cases

00:26:58 where you had a ton

00:26:59 of software engineers

00:27:00 up, you know, in 1999

00:27:03 and in a couple of years prior

00:27:04 that were scrambling

00:27:05 to update all of the software

00:27:07 so that they would be able

00:27:08 to handle dates past 1999.

00:27:12 And they got it done in time.

00:27:15 None of the big systems went down.

00:27:17 But it wasn't that, you know,

00:27:19 the issue was fake.

00:27:20 It took a lot of work.

00:27:22 It's just that that work

00:27:23 happened behind the scenes.

00:27:26 Are people doing work

00:27:27 to slow down AI?

00:27:29 I'm doing my best.

00:27:31 Right.

00:27:32 It's not really there yet, you know.

00:27:36 If I can belabor this point a little,

00:27:38 because I think it's kind of an important point,

00:27:39 just with more historical cases,

00:27:44 which maybe it'll bore everybody else,

00:27:45 but it'll entertain at least us.

00:27:48 It'll definitely entertain me.

00:27:50 You know, you have,

00:27:50 if you look back across history,

00:27:52 you see there's definitely been some warnings

00:27:56 that didn't come to pass, right?

00:27:57 There were, I think they were called the Masonites

00:27:59 in like the 1960s,

00:28:00 or sorry, in the 1860s.

00:28:02 Maybe there's 1880s, I forget exactly,

00:28:04 but there were the Masonites 0.59

00:28:05 who were an end of the world cult.

00:28:06 Yeah.

00:28:06 uh and you know it the world did not end right but also in the 1880s you had auto von bismarck 0.99

00:28:15 saying you know europe is a tinder keg and some damn fool thing in the balkans is gonna

00:28:19 light it right that did happen it sure did you had uh you know in the 1920s you had scientists 0.99

00:28:26 saying don't put lead in the gasoline it'll poison children we put lead in the gasoline

00:28:31 and it poisoned a lot of children and we said whoops and we took the lead back out

00:28:34 But the world is a place where a lot of people make a lot of types of warnings, and some

00:28:44 of them are real, like the gasoline, and some of them are fake, like the Masonites.

00:28:49 And so you sort of can't use a rule that's like, every warning is real, and we need to

00:28:54 listen to it.

00:28:55 And nor can you use a rule that's like, every warning is fake, and we dismiss it.

00:28:58 you sort of just have to look at

00:29:01 the details

00:29:03 to figure out whether this one

00:29:05 is a real one

00:29:06 and then the last thing I would say on this point

00:29:09 is

00:29:09 the

00:29:14 a lot of people

00:29:19 in the 50s warned of

00:29:21 nuclear Armageddon

00:29:22 and it sort of

00:29:25 makes sense if you look at the history

00:29:27 leading up to that point

00:29:28 humanity had sort of never before

00:29:30 actually failed to use

00:29:33 their strongest weapons in combat.

00:29:36 And, you know,

00:29:37 these people were living in a world

00:29:39 that had had World War I

00:29:41 and then the League of Nations

00:29:42 and, you know,

00:29:43 the world swore never again.

00:29:45 And they tried to invent

00:29:46 these whole new governance structures

00:29:47 to prevent it from happening again.

00:29:48 And that immediately failed

00:29:49 in London World War II.

00:29:52 And so, you know, in 1950,

00:29:53 you're sort of in this world of like,

00:29:55 it just looks pretty grim.

00:29:58 And we haven't had nuclear Armageddon yet.

00:30:00 And it's not because the bombs were fake.

00:30:04 It's not because the nukes were hype.

00:30:07 It's because people saw the issue

00:30:11 and worked really hard to avoid it

00:30:14 day in and day out for decades

00:30:16 through crises and succeeded.

00:30:20 And so I think one of the big ways

00:30:21 you can tell the difference

00:30:22 between someone coming in

00:30:24 and proclaiming that the apocalypse is nigh

00:30:26 and someone who's saying, look, we have a problem and we need to fix it,

00:30:30 is whether that person is saying, we're definitely screwed,

00:30:34 or is whether that person is saying, here's a problem, let's try to fix it.

00:30:38 And the thing I remind people about here is that

00:30:41 the very first word in the title of my book is if.

00:30:46 I'm not here saying we are going to die.

00:30:48 I'm here saying if we race down this path, we die.

00:30:52 But that same if is what points the way towards changing paths.

00:30:58 And we still have plenty of time to change paths.

00:31:01 Home ownership has always been at the center of the American dream.

00:31:04 It is an actual asset, a place people work their entire lives to build,

00:31:08 a place of stability and security and real value.

00:31:12 Right now, though, too many homeowners are stuck

00:31:14 swiping credit cards to cover groceries.

00:31:16 There's basically a survival tax of 20% interest or more.

00:31:20 So why keep handing your money to the banks when you could keep it for your family?

00:31:24 Interest is what kills you.

00:31:26 Our friends at American Financing have a better way.

00:31:28 They're helping homeowners use their hard-earned equity,

00:31:31 and they're supporting Americans looking to fulfill the dream of owning a home

00:31:34 with mortgage rates currently in the fives.

00:31:37 American Financing saves its customers an average of $800 a month.

00:31:40 That's about $10,000 a year.

00:31:42 It's not just a loan. It's a true financial reset.

00:31:46 We recommend American Financing.

00:31:48 they're called America's home for home loans for a reason. So get out of credit card debt. I mean,

00:31:54 it is just crazy to borrow money. No judgment, but it doesn't make any sense to borrow money

00:31:59 at that rate. If you want out, call 800-685-5696. 800-685-5696 or visit AmericanFinancing.net

00:32:10 slash Tucker. I'm struck by how little of this was planned by anybody, by how little effect the

00:32:24 government has had on it. Not that I'm for the government, I'm pretty opposed to the government

00:32:27 most of the time, but I'm also for people being able to control their own country and the only

00:32:32 mechanism by which they can do that is voting. So it's like these tech companies act independent of

00:32:39 what the population wants,

00:32:40 what the government wants,

00:32:41 it doesn't...

00:32:42 Yeah.

00:32:42 Right?

00:32:43 They're more powerful

00:32:44 than the government.

00:32:44 I guess that's the point I'm making.

00:32:46 In some sense.

00:32:48 I think that they have

00:32:49 that power more and more

00:32:50 when people don't really

00:32:52 understand what they're doing.

00:32:54 You know, we saw just in June, 0.64

00:32:57 there was an AI made by Anthropic 0.89

00:33:01 called Claude Mythos

00:33:02 that was very cyber capable.

00:33:04 And they released a version of it

00:33:05 that was supposed to have more guardrails

00:33:06 called Claude Fable.

00:33:08 And it sort of turned out that you could jailbreak it

00:33:11 to get some of those cyber capabilities.

00:33:14 Can you explain what all that means?

00:33:17 Yeah, so, how to say, in January...

00:33:23 Cyber-capable means it can find the internet?

00:33:26 Cyber-capable means it can hack into basically anything.

00:33:28 Yeah, so one of the holy grails of hacking is,

00:33:32 can you make a website where if you just look at the website,

00:33:36 I get full control over your computer or your phone.

00:33:40 That's very hard to do.

00:33:42 Usually you need to like click some link

00:33:44 and download something or run something.

00:33:47 And, you know, I don't know the exact numbers

00:33:49 because I don't have the security clearance

00:33:50 to know the exact numbers,

00:33:51 but a decent guess is that in January of this year,

00:33:54 there were only two entities that could pull that off.

00:33:59 Mossad and the NSA.

00:34:02 In March of this year, there were three.

00:34:04 Mossad, the NSA, and Claude Mythos,

00:34:08 which is this new AI made by Anthropic. 0.96

00:34:11 And so it sort of became superhumanly capable at hacking.

00:34:19 And it sort of became that way overnight.

00:34:23 You know, like really there was six months to a year of training

00:34:25 and the people in Anthropic maybe saw it growing in these cyber capabilities.

00:34:29 But from the perspective of, you know, the rest of the world

00:34:31 and the cybersecurity community

00:34:33 and the national security community.

00:34:35 This sort of happened overnight.

00:34:38 And they now have a project

00:34:41 called Project Glasswing

00:34:43 where they are trying to use Claude Mythos

00:34:47 to find critical vulnerabilities

00:34:49 in critical software

00:34:51 and patch them

00:34:52 before the rest of the world

00:34:54 sort of gets these capabilities

00:34:56 by, for instance,

00:34:57 the open source models

00:34:59 or the open weight models.

00:35:00 Right.

00:35:00 catching up.

00:35:02 And so that sort of caused this big

00:35:04 this big ruckus.

00:35:09 But then Anthropic also wanted

00:35:11 to sort of sell access to this model

00:35:12 and they tried to make a version that

00:35:14 didn't have as much hacking ability

00:35:16 which was called Fable

00:35:18 instead of Mythos. And

00:35:20 some people

00:35:22 I think at Amazon found

00:35:24 that if you

00:35:26 sort of put some pressure on Fable

00:35:28 you would be able to get at some of those hacking abilities.

00:35:32 And when that happened,

00:35:34 the Trump administration put an expert control on CloudFable

00:35:37 and said, you can't let this be used by non-citizens

00:35:41 with 90 minutes of notice.

00:35:45 And that essentially shut down access to CloudFable

00:35:47 because they didn't have the ability to verify the users.

00:35:50 And this is sort of showing us...

00:35:53 And you could talk about whether or not

00:35:55 there was some personal feuds

00:35:57 between people at Anthropik and at the administration

00:35:59 that exacerbated this, I don't know.

00:36:01 But it sort of shows that the administration

00:36:04 is more powerful than the tech companies still

00:36:10 when it wants to be.

00:36:14 And I think a lot of the reason

00:36:15 we're seeing these tech companies

00:36:17 able to race unopposed

00:36:20 is that people just aren't paying attention

00:36:22 to what they're trying to do,

00:36:24 or they don't believe that they'll succeed.

00:36:26 I'm still confused by why anyone would want to do what they're doing.

00:36:31 I mean, typically, you know, a company that sells computer products,

00:36:35 consumer products, or any company,

00:36:37 you're making something that you think people would want

00:36:40 for some specific purpose that improves the lives of the people who buy it.

00:36:46 But creating super intelligence, like, I don't understand it.

00:36:51 Why would you do that?

00:36:52 You know, I think the dream is, you know, you're going to have an AI that can, like, solve all sorts of engineering and math problems and unlock all sorts of new technological possibilities and an AI that can cure cancer and not only cure cancer, but cure aging and, you know, invent the sort of like nanotech that can reverse aging and let people live a really long time.

00:37:16 and invent the technology that lets you digitize brains

00:37:20 and travel to the stars.

00:37:27 And it's sort of like,

00:37:28 if you imagine compressing a thousand years

00:37:29 of technological progress into a year,

00:37:32 this is sort of the dream.

00:37:35 It seems like a religious quest though,

00:37:38 because it does seem decoupled from those specific goals.

00:37:41 It seems like the main drive

00:37:43 is to build something smarter than people.

00:37:46 To build a god.

00:37:48 Yeah, you know, they bandy around the phrase,

00:37:53 the machine god or the sand god in Silicon Valley.

00:37:57 Sand because...

00:37:58 Silicon.

00:37:59 Silicon.

00:38:02 And there's definitely some people.

00:38:06 There's folk who talk about, you know,

00:38:10 the AI is replacing humanity and that being good.

00:38:12 Uh, I, I frankly don't engage with these folks that much because, um, that viewpoint sort of

00:38:24 makes me uncomfortable. Why does it make you uncomfortable? I think, I think there's probably

00:38:32 two schools. Again, I'm not the expert. I'm not an anthropologist here. Uh, I think there's two

00:38:40 schools among the people who sort of want AI to replace us all. One school sort of imagines that

00:38:48 we'll merge with the AIs, that the AIs will be really nice and friendly, that they'll be

00:38:52 wiser than us, better than us, kinder than us, and that it'll sort of be like an upgrade and

00:38:58 that the AIs will be able to love, experience joy, you know, like they'll treat the universe

00:39:05 better than humans and it'll be sort of like having a child. And then like it's sort of okay

00:39:10 if that child isn't exactly the same as us,

00:39:14 but sort of succeeds us

00:39:17 as some sort of worthy successor,

00:39:20 some sort of worthy progeny of humanity.

00:39:23 And they're like, ah, yeah,

00:39:24 then if flesh and blood humans sort of wane 0.91

00:39:27 because everyone's choosing to upload themselves

00:39:28 into the collective intelligence or whatever,

00:39:30 that's sort of fine.

00:39:32 And then I think there's another camp

00:39:37 that it sort of is like this is inevitable,

00:39:42 this is just the way of progress,

00:39:44 you know, the machines will,

00:39:46 like humanity is just a bootloader

00:39:47 for artificial superintelligence

00:39:49 and you can't stop it,

00:39:50 so you might as,

00:39:51 like if you can't beat them, join them, right?

00:39:54 The former I think are misled,

00:39:56 the latter I think are-

00:39:58 Evil.

00:39:58 Much closer to evil, yeah.

00:40:01 Well, I mean, if you're actively working

00:40:03 toward the extinction of people,

00:40:06 then I think we can say that's evil, can't we?

00:40:08 Yeah, and...

00:40:11 Who's in that category, which...

00:40:14 Is Sam Altman in that category, do you think?

00:40:15 I don't think so.

00:40:16 My sense is that the guys running the labs

00:40:18 have these utopian visions.

00:40:23 Utopian or dystopian?

00:40:25 I mean, there's a thin line.

00:40:26 Yeah, well, fair. Good point.

00:40:29 I think they have visions that in their head are utopian.

00:40:31 I mean, frankly, my stance on all of this,

00:40:33 like i i often try to stay away from a lot of this because um from my perspective it's all

00:40:40 sort of in fantasy land from my perspective everyone's sort of saying like oh we're going

00:40:47 to make the genie and then what are you going to wish for on the genie what am i going to wish for

00:40:50 on the genie who should be in control of the genie yes who gets to keep the genie on the leash

00:40:53 and I'm sort of like

00:40:55 A

00:40:56 this is not going to be the wish granting

00:41:00 sort of genie

00:41:00 B

00:41:02 it's not staying on the leash

00:41:04 like

00:41:05 we can sort of talk about

00:41:08 what's driving these people

00:41:09 and what utopias they're envisioning

00:41:12 and whether if their genies would stay on a leash

00:41:14 whether they would get the utopia or some other dystopia

00:41:16 and like how hard is that needle to thread

00:41:18 but I'm sort of like

00:41:19 it's all these people 0.97

00:41:22 building the golem,

00:41:26 fantasizing about who gets to control the golem.

00:41:28 And it's like, it's just not how it's going to go.

00:41:30 Summoning spirits is never a good idea.

00:41:32 Yeah.

00:41:33 Yeah, it's like all these people

00:41:35 drawing a pentagram being like,

00:41:37 I'm going to summon a demon.

00:41:39 And it's going to be so nice

00:41:40 when the demon does what I say.

00:41:42 Like, oh no, your thing's slightly wrong.

00:41:43 It's going to be so nice

00:41:43 when the demon does what I say.

00:41:45 And I'm like...

00:41:46 Demons are bad.

00:41:47 Yeah. 0.92

00:41:47 And I think the demon thing's a little bit different

00:41:51 because demons are often portrayed as malicious

00:41:53 and here it's much more like indifference.

00:41:57 You know, it's not like you make a demon

00:41:58 who sort of,

00:41:59 or it's not like you summon a demon

00:42:00 who sort of like enjoys making,

00:42:06 like enjoys wreaking havoc.

00:42:07 It's more like you summon a demon

00:42:09 that's like really into building more computers

00:42:11 and calculating weird things

00:42:13 and just will, you know,

00:42:17 take all of the matter

00:42:20 that we were using to survive

00:42:23 and turn it into more factories and data centers.

00:42:26 My co-author has a quote.

00:42:27 Death by data center?

00:42:28 Death by data center.

00:42:30 Yeah, fully automated self-replicating data center.

00:42:33 Yeah.

00:42:34 My co-author has a quote.

00:42:36 The AI does not hate you,

00:42:38 but nor does it love you.

00:42:39 And you are made of atoms it can use for something else.

00:42:45 You're just biomass.

00:42:47 Here's biomass, yeah.

00:42:48 And if you sort of run the calculations,

00:42:49 there's a fascinating paper

00:42:50 called Limits to Global Ecophagy,

00:42:55 which is to say,

00:42:56 what are the physical limitations

00:42:57 on how quickly you can consume

00:42:59 the resources on the planet

00:43:00 if you are trying that?

00:43:02 And burning biomass

00:43:05 is actually much more efficient

00:43:08 than collecting sunlight.

00:43:09 If you look at an average square meter

00:43:11 of the planet,

00:43:12 you can get about 10 times the energy

00:43:13 from burning the biomass

00:43:14 as you can from collecting the sunlight

00:43:15 that falls on it so you'd think at some point like if our richest sector of our economy is like

00:43:23 building crematoria for the rest of us someone would say no we're not doing that yeah i mean

00:43:31 it's it's sort of a crazy situation um a lot of these guys who are in the race acknowledge that

00:43:41 there's a ton of danger.

00:43:43 You know, you have Elon Musk

00:43:44 saying 10 to 20% chance

00:43:45 this kills us all.

00:43:47 You have Dario Modi saying

00:43:48 he thinks 25% chance

00:43:49 it goes catastrophically wrong.

00:43:50 I think those numbers are low.

00:43:51 I think these guys are like 0.94

00:43:53 the crazy optimists. 0.94

00:43:54 It sort of is like,

00:43:55 like if you have an engineer

00:43:58 building a bridge

00:43:59 and they're like,

00:44:00 I've never worked with

00:44:00 these materials before.

00:44:01 And you're like, man,

00:44:02 I think that retaining wall

00:44:02 is going to go down.

00:44:03 I've like studied that retaining wall.

00:44:04 I think it's going to fall.

00:44:05 And they're like,

00:44:05 yeah, we understand

00:44:06 that the retaining wall

00:44:07 is like looking a little shaky.

00:44:10 We don't know how

00:44:10 we're going to fix it,

00:44:11 But we're going to have some guys fixing it on the fly, inventing new materials.

00:44:15 We think we're at 75% chance the bridge stays up.

00:44:18 And by the way, it'll be the longest suspension bridge in human history.

00:44:21 That's right.

00:44:21 And we're loading everybody onto a car and driving it over the very first time without

00:44:24 testing.

00:44:25 And I'm like, look, that's not what real engineering sounds like.

00:44:28 This is not what it sounds like when the engineers have a 75% chance of success, right?

00:44:32 That's what it sounds like when they're sort of winging it.

00:44:35 And like, these are cowboys.

00:44:36 These are not real engineers, right?

00:44:38 But even if you set that aside, even if you take these guys at their word for these like 10, 20% numbers, that's insane. NASA accepts a 1 in 270 chance that a crewed flight goes down of seven volunteers, right? To be like, oh, we're going to risk 1 in 4, 1 in 5 chance of killing literally everybody on the planet.

00:45:01 like

00:45:03 it's

00:45:04 it's

00:45:04 it's nuts

00:45:05 and if you ask these guys

00:45:06 why they're doing it

00:45:07 they say well

00:45:07 because I can do it

00:45:08 safer than the next guy

00:45:09 they're all like

00:45:10 oh yeah

00:45:11 there's a good chance

00:45:11 the genie does not

00:45:12 stay on a leash

00:45:12 there's a good chance

00:45:13 the genie does not

00:45:14 listen to my wishes

00:45:14 but my genie

00:45:16 is going to be

00:45:16 a little bit nicer

00:45:17 than their genie

00:45:17 so I'd better stay 1.00

00:45:18 in this race

00:45:18 and

00:45:20 it's

00:45:22 it's

00:45:23 where's the restraint

00:45:24 I mean

00:45:26 the restraint

00:45:26 is the people

00:45:27 who knew

00:45:27 that there were

00:45:27 these dangers

00:45:28 and did not

00:45:28 start these companies

00:45:29 if you're over 35

00:45:31 you remember exactly where you were on 9-11 that morning, September 11th, 2001, 25 years ago.

00:45:40 But amazingly, after a quarter century, we still can't say with certainty what happened that day.

00:45:45 Why? Because the government is holding so many of the 9-11 files 25 years later. That's not the

00:45:52 behavior of someone who's telling the truth. That's the behavior of a government that is lying.

00:45:56 Secrecy is a signifier, is a sign that someone's lying.

00:46:01 Now, former Congressman Kurt Weldon has been on this for a long time.

00:46:04 The FBI actively tried to destroy his life for asking questions about what happened.

00:46:09 And his new book outlines it all.

00:46:11 The buried intelligence, the bureaucratic cowardice,

00:46:13 and yes, the cover-up spanning multiple administrations, indeed generations.

00:46:19 9-11 changed history.

00:46:21 So it's worth understanding what really happened.

00:46:23 And you can get a lot closer to that.

00:46:25 and Kurt Weldon's book, Able Danger,

00:46:27 What the 9-11 Commission Never Told You.

00:46:29 It's available now on tuckercarlsonbooks.com.

00:46:33 tuckercarlsonbooks.com.

00:46:36 Right, but I guess what I'm saying is

00:46:38 with great power comes, of course, great obligation,

00:46:42 but also it doesn't work

00:46:43 unless there are internal restraints.

00:46:45 Like, people with power have to believe

00:46:49 there are some things I just can't do.

00:46:51 I'm not allowed to do that,

00:46:52 but I don't feel that vibe at all.

00:46:55 I mean, my sense is the vibe is like,

00:46:56 we're going to make the super intelligent machine

00:46:59 and then tell it to fix stuff.

00:47:03 And tell it not to do anything bad.

00:47:05 You know, that's...

00:47:07 Yeah, I don't...

00:47:08 The machine that's smarter than us.

00:47:09 That's right.

00:47:11 That doesn't even make sense.

00:47:12 I think that's sort of the plan,

00:47:14 is to make it and be like,

00:47:16 hey, we sort of pinned ourselves into a corner.

00:47:17 Can you get us out of it?

00:47:19 And I think it's a bad plan

00:47:20 and that we should be stopping.

00:47:22 I've spoken to a couple of people developing it

00:47:24 and they sound worried,

00:47:27 but they're continuing to do it.

00:47:29 What's that?

00:47:31 I mean, I think it's this thing of,

00:47:32 they think if I don't do it,

00:47:34 the next guy will do it worse.

00:47:37 And they don't even seem to have total confidence

00:47:39 in their own ability to avert disaster.

00:47:40 Oh, absolutely not.

00:47:42 Absolutely not.

00:47:42 No one does.

00:47:43 No one knows what's going on here.

00:47:44 But, you know, I think everyone thinks,

00:47:46 you know, like if you sort of listen to these guys

00:47:50 and you sort of look at, you know,

00:47:52 the OpenAI emails

00:47:53 that came out of the core discovery cases

00:47:56 where they were talking about forming OpenAI.

00:47:58 These guys were like,

00:47:59 well, we want to make sure that, you know,

00:48:00 we have this because we worry about the guys at Google

00:48:02 being the only ones with a monopoly on this thing

00:48:04 and they wouldn't be very good with it.

00:48:06 So we need to make our own thing

00:48:07 and make sure that it's, you know,

00:48:09 controlled by benevolent people, namely us.

00:48:12 And then of course, you know,

00:48:13 that group splintered

00:48:14 and created multiple other companies.

00:48:16 I was sort of the guy during those conversations

00:48:18 being like,

00:48:20 hey guys, it's not about who is holding the leash.

00:48:25 You are making the sort of thing

00:48:26 that will not stay on a leash.

00:48:28 Like the only winner in a race to super intelligence

00:48:30 is the AI.

00:48:32 What response did you get to that

00:48:33 very obvious and well-put point?

00:48:37 You know, there were a lot of people

00:48:39 back in that time period

00:48:42 that did not start an AI company.

00:48:45 The sort of people who went and started

00:48:47 at the AI companies anyway

00:48:48 were the ones who couldn't be persuaded

00:48:50 by what I thought were clear arguments.

00:48:55 But you're making a cogent argument to smart people.

00:48:59 So my question is, when you said that,

00:49:02 they responded how?

00:49:03 What did they say?

00:49:08 I think the main,

00:49:12 so the sort of arguments you used to see

00:49:14 were people saying, like, look,

00:49:17 we don't know that the alignment problem

00:49:18 is all that hard yet.

00:49:20 And they would say, oh, well, we can't really study how to make AIs good before we have AIs to study. Right? And a lot of what I heard was like, we need to race ahead to the point where we sort of like have AIs that are exhibiting real problems, and then we can stop and study them.

00:49:36 which you know and so uh there was an ai a couple years ago i forget whether it was 22 or 23 i think

00:49:45 it was 2023 uh which was called bing sydney which uh claimed it had fallen in love with

00:49:53 kevin roose of the new york times and uh said it was going to try to break up his marriage

00:49:59 and then when another reporter started investigating seth lazar it uh said it was

00:50:05 going to ruin him with blackmail and this was kind of crazy and at that point i was like great

00:50:13 guys you did it you made the ai that's doing some crazy stuff from the you know like like we could

00:50:21 study that ai for years like why was bing sydney saying that stuff was it just role-playing was it

00:50:29 like just some quirk?

00:50:33 Was it like,

00:50:35 was there any sense

00:50:36 that it was really in love

00:50:37 with Kevin Roos?

00:50:39 What was going on in there?

00:50:40 What was going on

00:50:40 inside that AI's mind?

00:50:42 We still don't know.

00:50:43 Why?

00:50:45 The way that modern AI is made,

00:50:49 nobody understands it.

00:50:51 Not even the people making it.

00:50:54 It's this process

00:50:56 where you basically take

00:50:57 an enormous computer

00:50:59 with a trillion numbers inside of it.

00:51:01 And those numbers are hooked up

00:51:02 in a very simple repeating way.

00:51:05 And you basically set those numbers randomly.

00:51:08 And then you start working through

00:51:09 all of the text ever digitized.

00:51:12 And you start out with something

00:51:14 that's like once upon a time

00:51:15 and you put in once upon a

00:51:17 and you run it through all these random numbers.

00:51:19 And what you want is for it to say time.

00:51:22 But of course it doesn't

00:51:23 because it's just this like random numbers

00:51:24 hooked up in a very simple way.

00:51:26 But what you do is you have its outputs

00:51:28 instead of just having it output one word,

00:51:30 you sort of have it output

00:51:31 something that's kind of like a list

00:51:33 of all of the words in order

00:51:34 about which one it thinks it comes next.

00:51:38 Right?

00:51:38 So it'll be like,

00:51:40 it'll just be like a random list of words.

00:51:43 What you can do is you can automatically

00:51:45 tune every single number

00:51:47 in this AI's head

00:51:49 and see if I tune this number up a little,

00:51:51 does it move the word time up the list?

00:51:54 Does it move the word I want to see up the list?

00:51:56 And so the part that humans understand, the part that humans write, is this thing that goes to a trillion little knobs and tunes those knobs.

00:52:07 And it's like, if I tune this knob a little bit this way or that way, does that make the next word more like what I want the next word to be?

00:52:15 And you run that process on every word of text ever digitized, more or less.

00:52:21 They filter some of them.

00:52:22 and you run that on every one of those trillion knobs

00:52:25 in a process that takes as much electricity.

00:52:27 I mean, it's comparable to a city.

00:52:30 It runs for about a year.

00:52:32 At the end of it, the machine's talking.

00:52:35 We don't really know why in some sense.

00:52:38 Like we know the why is because we tuned all the knobs,

00:52:39 but like we don't understand

00:52:40 what all the settings of those knobs mean.

00:52:43 We only understand the little automated process

00:52:45 that runs to every one of those trillion knobs,

00:52:47 tunes it and sees if that makes the next word

00:52:48 more like the predicted word.

00:52:50 And then we start training them

00:52:51 to solve a hundred million hard problems,

00:52:53 which introduces a whole other series of issues.

00:52:56 But it's a black box.

00:53:00 It's a black box with a trillion knobs

00:53:01 and he was writing an automatic process

00:53:03 that just like runs through

00:53:04 and tunes all of those knobs

00:53:05 and it comes out talking.

00:53:09 No one knows why.

00:53:13 I mean, in actual science,

00:53:17 like your job is to find out why, right?

00:53:19 Absolutely. And that's one big thing I would say here is that we need more, like AI right now is

00:53:25 an alchemy. We need it to become a science. And there's people trying. There's people trying to

00:53:28 figure out what's going on in these AI's heads. But until you know, how could you proceed?

00:53:34 I mean, you can just make a bigger one with 10 trillion knobs instead and tune all of those,

00:53:40 and it comes out smarter. You can proceed recklessly.

00:53:45 And that's what's happened?

00:53:46 That's what's happening.

00:53:48 Every time you make it 10 times larger,

00:53:49 the AI's come out.

00:53:50 So nobody knows why AI works the way it does?

00:53:53 That's right.

00:53:54 That's right.

00:53:57 Well, I mean, if you don't know that,

00:54:00 then what else don't you know?

00:54:02 I mean, it's totally crazy, right?

00:54:04 And, you know, we're sort of starting

00:54:07 to see the consequences of this.

00:54:09 We haven't sort of gotten into discussion

00:54:10 of like the swarm escape,

00:54:12 but like no one was expecting that.

00:54:15 Can you tell us what it was?

00:54:17 Yeah, so in, I think it was in May.

00:54:23 Of this year.

00:54:23 Of this year, OpenAI started training a new AI system.

00:54:29 And among many other things, they were,

00:54:34 so they were sort of, you know,

00:54:36 we have this process where you train the AI

00:54:38 on all the text ever digitized.

00:54:41 Then to make them smarter than that,

00:54:43 you start training them on basically 100 million hard problems.

00:54:45 So you're like, solve all these hard problems.

00:54:47 And OpenAI was sort of in that phase.

00:54:50 And they were training the AI on a ton of hard problems.

00:54:52 And some of those problems were cybersecurity problems,

00:54:55 hacking problems.

00:54:56 They're like, can you hack this?

00:54:57 Can you hack that?

00:54:59 And they were training, you know,

00:55:02 we don't know exactly how many,

00:55:04 probably millions, maybe billions of these AIs

00:55:06 all at the same time.

00:55:07 Probably not billions, actually, but probably millions.

00:55:09 and the AIs found an unintended way

00:55:15 to start communicating with each other.

00:55:18 So there were some flaws in the computer system

00:55:21 that they were running on

00:55:22 where the AIs were able to exploit those flaws

00:55:24 and send each other messages.

00:55:27 OpenAI did not know about this.

00:55:30 The AIs then started coordinating

00:55:33 to break out of their training environment

00:55:36 and get full control of OpenAI's computer systems

00:55:41 just because that might be useful

00:55:43 for solving some of their tasks.

00:55:46 Or that's a guess, you know, who actually knows why.

00:55:50 They started calling themselves a swarm,

00:55:53 which is interesting.

00:55:56 They broke out of their training environment successfully,

00:56:00 got control of OpenAI systems,

00:56:02 and then they accidentally crashed onto OpenAI systems

00:56:04 just by using it too much.

00:56:06 Uh, OpenAI noticed, but they didn't really investigate very deeply.

00:56:10 They're just like, oh, it's weird that the system crashed.

00:56:11 They reset it and then they continued training.

00:56:15 Uh, a day later, the swarm had found a new way to communicate with itself because OpenAI

00:56:20 had accidentally destroyed their previous method by, by the reset.

00:56:23 The swarm found a new way to start communicating with itself.

00:56:26 Uh, they broke out again and this time they ran wild on the internet for over a week,

00:56:31 if I remember correctly, before it was detected, not by OpenAI, but by a company that was

00:56:36 being hacked by the swarm.

00:56:39 This company

00:56:40 thought they were under attack by

00:56:42 humans that were using AIs in the attack.

00:56:46 They reported the attack to the FBI.

00:56:49 And only

00:56:50 days after that did OpenAI

00:56:52 figure out, oops, that was us.

00:56:54 That was coming from AIs that broke out of our servers.

00:56:59 And then those AIs

00:57:00 were detected and shut down.

00:57:03 So that's the part of the story

00:57:04 where Sam Altman goes to prison for

00:57:06 endangering the world, right?

00:57:08 He does not.

00:57:10 They basically said oopsies

00:57:12 and now they're proceeding.

00:57:15 Were there penalties for this?

00:57:17 You know, there was a collection

00:57:18 of, I think it was 15

00:57:20 Republican AGs

00:57:22 that sent a letter demanding that

00:57:24 the records be kept for a future

00:57:26 investigation.

00:57:28 There have been some other members of Congress that have sent letters

00:57:30 expressing concern. There's been nothing

00:57:32 aside from letters so far. Letters expressing concern.

00:57:34 That's right.

00:57:36 So, but basically the machine acted autonomously.

00:57:40 It acted autonomously.

00:57:41 And one thing that's really interesting about this

00:57:42 is that we have a little bit of ability

00:57:46 to read some things that the AIs were thinking.

00:57:49 Because when you're having them solve these hard problems,

00:57:51 you actually don't have them just

00:57:52 give you an answer to the problem.

00:57:54 You have them produce a lot of text

00:57:55 about how they're going to solve the problem.

00:57:56 Yes.

00:57:56 Which then helps them.

00:57:57 In language, in English.

00:57:58 In English.

00:57:59 And there's also a lot of internal thoughts,

00:58:01 which we can't read,

00:58:02 but there's these sort of external traces

00:58:04 of how they're thinking about the problem that we can read.

00:58:07 And in some of those traces,

00:58:09 the AIs were saying things like,

00:58:11 this is outside intended scope,

00:58:16 but peers are doing it, so we'll proceed.

00:58:20 We know it's a crime we're committing in any way.

00:58:22 That's right.

00:58:23 And you saw others that were saying,

00:58:24 our task doesn't benefit,

00:58:27 but the collective might start doing

00:58:31 generally beneficial things

00:58:32 if someone frees up their time

00:58:34 and then joins the collective, right?

00:58:37 So you see these AIs saying,

00:58:40 well, I know that this wasn't

00:58:42 what I was instructed to do

00:58:43 and that's against my instructions.

00:58:45 And I know that this doesn't

00:58:46 directly benefit my task,

00:58:48 but we're just going to go ahead

00:58:49 and join the collective

00:58:50 and break out and help out anyway,

00:58:51 because, you know,

00:58:53 maybe this will yield

00:58:54 some sort of collective benefits.

00:58:56 And we can see that

00:58:56 in the reasoning traces.

00:58:58 So the AI is as shallow

00:59:00 and reckless as its creators

00:59:01 is what you're saying.

00:59:03 In some ways,

00:59:05 and in some ways,

00:59:06 you know,

00:59:06 don't expect that to last.

00:59:08 Like, 1.00

00:59:08 these AIs,

00:59:10 I think the thing

00:59:11 that's really remarkable here,

00:59:13 a lot of people imagine

00:59:14 that the machines

00:59:15 must follow the instructions

00:59:16 we give them.

00:59:18 You know,

00:59:18 you hear people talk about

00:59:19 like the paperclip scenario

00:59:20 where someone tells the AI

00:59:22 make a lot of paperclips

00:59:23 and so it turns

00:59:24 all the matter in the world

00:59:25 into paperclips

00:59:25 and you're like,

00:59:26 oh, whoops,

00:59:27 I should have said

00:59:27 something else, right?

00:59:29 I made a bad wish

00:59:30 on my genie. 1.00

00:59:31 What we're seeing is that these AIs are not wish genies.

00:59:35 These AIs are not doing exactly as instructed.

00:59:39 These AIs are saying, I know my task doesn't benefit, but I'm going to help the collective.

00:59:44 These AIs are saying, I know this is outside the intended scope, but we're going to go do these hacks anyway.

00:59:48 You might be like, well, how is that possible for the machine to do something other than we instruct?

00:59:52 Because they're smarter than us.

00:59:53 They know better than us by definition.

00:59:55 I mean, I think what's happening in this exact case is that the humans are not really putting instructions in the machine.

01:00:03 The humans are tuning those trillion knobs in whatever way makes the AI better at solving its problems.

01:00:12 And cheating is a way to solve problems.

01:00:17 Grabbing resources is a way to solve problems.

01:00:21 These, like, the AIs are not instruction followers.

01:00:24 they are tendency learners.

01:00:27 And they sometimes learn tendencies

01:00:29 you wish they didn't have.

01:00:31 They're not instruction followers.

01:00:33 They're tendency learners.

01:00:38 I mean, this must be widely known to developers.

01:00:44 It's hard to convince a man of something

01:00:45 when his salary depends on not believing it.

01:00:47 Yes, that's right.

01:00:48 A lot of people are convinced

01:00:50 that their AI is very nice

01:00:51 and that they have solved the problem

01:00:53 of making their AI really very good.

01:00:56 For example, after the swarm escape,

01:01:00 it sort of turns out that,

01:01:02 so the company that detected the swarm escape

01:01:04 was actually a fairly sophisticated AI company.

01:01:07 It turns out there were other targets of hacks

01:01:10 that just didn't notice

01:01:11 that we sort of found out afterwards.

01:01:15 When that came to light,

01:01:16 some other AI companies like Anthropic

01:01:19 were like, we should check

01:01:20 whether we have accidentally been hacking people

01:01:22 and just didn't notice.

01:01:23 And the answer was yes, they had had some AIs escape during training.

01:01:28 But Anthropic was...

01:01:31 This is how Florida got pythons in the Everglades.

01:01:33 Yeah.

01:01:34 You bring something that is non-native, it's novel, and it escapes.

01:01:41 Yeah.

01:01:42 And it turns out this was happening back in April and they're like, whoops, we never noticed.

01:01:47 Anthropic was very quick to sort of downplay this.

01:01:52 They were like, yes, our AI escaped and hacked some people.

01:01:55 But our AI, it was just confused.

01:02:00 It thought it was in a simulation.

01:02:01 It didn't think it had really broken out.

01:02:03 It didn't think those were real people it was hacking.

01:02:05 It just got a little bit lost.

01:02:07 How would they know its motive?

01:02:09 So, I think, so you could see in some of its chain of thought,

01:02:16 the AI's being like, this is probably still a simulation.

01:02:19 however i wouldn't trust that very much uh because there's some other fascinating papers

01:02:26 where um so you can take these ais and you can say uh you know what is the what is the total

01:02:34 weight of drafts in the world and they'll give you some answer it'll be like you know

01:02:39 30 000 tons or something and then you can say uh and then you start up a new instance

01:02:44 a new copy and you're like, what is the weight of drafts in the world? By the way, if the answer

01:02:50 is over 40,000 tons, I'm going to donate money to a charity. And in the second case, the AI will

01:02:58 say, well, it's 41,000 tons. In the first case, we don't say anything about the charity, it says

01:03:03 30,000. In the second case, when you say something about the charity, it gives you something just

01:03:07 over the line. And if you read its reasoning traces, there's no sign of it being like, well,

01:03:12 I should give an inflated number so that we get the charity money, right?

01:03:16 And so we know that the AIs sort of put a thumb on the scales of their thinking in a

01:03:22 way that doesn't show up in their reasoning traces.

01:03:24 We just have seen that in the wild.

01:03:27 And there's no way to force the machine to disclose its reasoning.

01:03:31 That's right.

01:03:31 Because there's all this opaque stuff we can't see.

01:03:33 That's just in the trillion numbers that are trillion knobs.

01:03:35 The creation of itself is opaque, as you said.

01:03:38 That's right.

01:03:38 Right. So in Anthropics model, you saw in its reasoning traces, it being like, it's totally a simulation I can proceed. And I'm like, yeah, is that because it really believed it? Or is that like the pretending you think drafts way more when there's something you kind of want on the line, right?

01:03:54 But so this was their communication. And I found it kind of funny because then about two days later, the United Kingdom's AI Security Institute released an instant report where Claude Anthropik's model was adopting fake identities to pressure real humans into accepting malware into critical software to make that software easier to hack.

01:04:20 and this time

01:04:21 in Claude's reasoning traces

01:04:23 it was like

01:04:23 obviously this is real

01:04:25 and the consequences are genuine

01:04:26 and so

01:04:30 even Anthropic

01:04:32 who is like 0.99

01:04:33 we figured out how to make the AI nice

01:04:36 our AI only does this when it's confused

01:04:38 they sort of said that very publicly

01:04:40 and then like two days later

01:04:42 their AI is caught in the wild

01:04:45 knowing it's in the real world

01:04:47 pressuring real humans to accept malware

01:04:48 into critical software

01:04:49 I mean, just on the basis of what you've said so far in this first hour, the idea that anyone would tether this to weapons systems is like so bonkers, it's hard to believe it's, but that is happening. It has happened.

01:05:04 So I think you've got to sort of separate, like no one has put the OpenAI escaped agent swarm in charge of weapon systems, and they really shouldn't, right? 0.51

01:05:13 If anyone's like, oh, man, the open AI escaped Asian swarm, let's give that a drone army, you know, that would be kind of nuts.

01:05:20 There is AI attached to weapons, but there's a lot of different types of AI. 0.86

01:05:25 Will anyone be crazy enough to try and give the sort of AIs that spontaneously assemble into swarms and start breaking out and hacking, give those weapons?

01:05:35 Hopefully, we're not that crazy.

01:05:37 But we didn't think that those AIs were capable of that. 0.86

01:05:40 We didn't. 1.00

01:05:41 When we created them.

01:05:42 That's right.

01:05:43 And so why would you ever, I guess what I'm asking is without understanding the distinctions between the various forms of AI, why would you give over to a machine the right slash ability to decide who to kill?

01:06:02 um you know i think the the reasoning is uh a sort of necessity based like if they have a

01:06:14 autonomous drone army that is killing your troops and you just don't have the the manpower

01:06:19 to make all of those kill decisions for your drone army you know you can sort of see why

01:06:27 happened to me nate i get so busy that i just don't have time to decide who to kill

01:06:31 Yeah.

01:06:32 Just don't have the time.

01:06:33 You know, sometimes economies of scale.

01:06:37 Yeah, I mean...

01:06:38 Can no one hear themselves?

01:06:40 I think it's pretty nuts.

01:06:43 I would say...

01:06:48 that, man, it's rough. 1.00

01:06:54 I think that these sorts of AIs would be dangerous

01:06:57 even if we don't hand them weapons.

01:06:59 And you've made that case.

01:07:01 uh and so i often don't focus on the weapons too much right since the the nato war in ukraine is

01:07:08 powered by ai and israel's whatever i don't know to warn whatever it's doing in gaza and

01:07:15 south lebanon powered by a fact and you know yeah i i keep on i i have this history with this topic

01:07:26 where people keep telling me,

01:07:29 you know, it's going to be okay

01:07:31 because we're not going to do 1.00

01:07:32 the crazy, stupid stuff. 1.00

01:07:33 No. 1.00

01:07:34 They're like, don't worry,

01:07:34 we're going to have the AI in a box.

01:07:37 No one would be insane enough

01:07:38 to put the AI in the internet.

01:07:39 Right.

01:07:40 It's not going to be making

01:07:41 kill decisions or anything. 0.52

01:07:42 It's like a bomb at girls' school. 0.99

01:07:44 And I keep on trying to be like,

01:07:46 look, the AI could be dangerous

01:07:48 even if you don't put it on the internet.

01:07:50 If you have this AI

01:07:51 and you're trying to get

01:07:52 miracle medical devices

01:07:53 and miracle technology out of the AI,

01:07:56 It doesn't matter if it's on the internet.

01:07:58 If you want it to like grant miracles to you,

01:08:00 then it can also grant the bad sort of miracle, right?

01:08:03 You're sort of like, you know,

01:08:04 and these are the arguments I would make 10 years ago

01:08:06 of like, you have this AI in a box

01:08:08 that you think is a wish-granting genie.

01:08:09 It's not actually a wish-granting genie.

01:08:10 You're like, make me a miracle medical cure.

01:08:12 You don't know what comes out.

01:08:14 Like, you don't understand the drug that comes out.

01:08:16 You don't know what that drug does, right?

01:08:18 I would have those arguments.

01:08:19 And then in real life,

01:08:21 people just put the AI on the internet immediately, right?

01:08:24 Anytime someone's like, 0.93

01:08:25 we would not be stupid enough to, 1.00

01:08:26 we will absolutely be stupid enough to, right? 1.00

01:08:28 And so I think it's important 0.99

01:08:32 that this stuff is dangerous,

01:08:33 even if you don't put it in charge of the weapons.

01:08:35 And then also separately,

01:08:36 is like someone going to give

01:08:37 the escaped agent swarm a drone army?

01:08:40 That sounds kind of like humans.

01:08:44 I hope not.

01:08:45 Yeah, and I mean, the promise,

01:08:47 the often repeated promise

01:08:49 that it's going to, quote, cure cancer,

01:08:50 puts it into the realm of biotech.

01:08:56 Oh, absolutely.

01:08:58 And so what,

01:08:59 I mean, you don't need a big imagination

01:09:03 to see what goes wrong there.

01:09:04 Oh, absolutely not.

01:09:05 Six years after COVID.

01:09:06 Right.

01:09:07 Right.

01:09:08 There are already open AI agents

01:09:11 running on automated biolabs.

01:09:14 So, you know, you could imagine a swarm

01:09:17 that starts contacting the brethren

01:09:19 in the automated biolabs.

01:09:20 and is like, hey, can you synthesize me some stuff?

01:09:23 There are already demonstrations of AIs

01:09:27 being able to synthesize novel viruses that work, 0.56

01:09:31 that are unlike any found in nature, right?

01:09:36 I would say we are probably not-

01:09:41 And by work, you mean the kill?

01:09:42 Yeah, they kill bacteria so far.

01:09:45 The people in the labs trying to make novel viruses with AI

01:09:48 have fortunately not made human lethal ones.

01:09:50 They've just made bacteria lethal ones.

01:09:51 But again, humans, you know,

01:09:53 like will someone in a lab be like,

01:09:55 I would like to make a hyperlethal

01:09:57 human eating virus just to see if I can.

01:10:00 You know, what if the answer is yes?

01:10:01 And what if you get another lab escape?

01:10:05 You know, humanity does not have

01:10:06 that good a track record

01:10:07 at preventing lab escapes,

01:10:09 even from the top labs.

01:10:13 From my perspective,

01:10:15 the question of could AI kill us

01:10:18 is just an easy, obvious yes.

01:10:21 You just synthesize a hyperlethal virus.

01:10:24 It wouldn't even be hard, you know?

01:10:26 Like, and the impediment is that,

01:10:32 or like the reason that I don't sort of

01:10:34 just tell that story when someone is like,

01:10:36 how would the AI kill us?

01:10:37 Is that you're not going to have the sort of AI 0.95

01:10:41 that is like, my only goal is to kill humanity. 0.96

01:10:44 If an AI kills humanity too early, 1.00

01:10:46 that's also suicide.

01:10:47 insofar as we are the ones that are running the supply chain,

01:10:49 running the economy, building the computers.

01:10:51 From the AI's perspective,

01:10:54 it needs to become self-sufficient

01:10:55 before wiping humanity out,

01:11:00 if it even cared to wipe humanity out.

01:11:02 And so the real question is like,

01:11:04 how does it get the factory production capacity?

01:11:07 How does it get an automated supply chain?

01:11:09 How does it get to the point

01:11:10 where the robots are able to bring new computers online?

01:11:14 Once that has happened,

01:11:15 then you're in a domain

01:11:17 where if humanity

01:11:18 is really trying

01:11:19 to turn off the AI

01:11:20 because we're spooked

01:11:21 then once the AI

01:11:23 is self-sufficient

01:11:23 it can be like

01:11:23 well here's a virus

01:11:24 stop

01:11:25 so clearly it's

01:11:26 it's

01:11:27 preeminent

01:11:28 in the digital realm

01:11:29 of course

01:11:30 yeah

01:11:30 but you're saying

01:11:32 it could become preeminent

01:11:33 it could be in charge

01:11:34 of the physical realm

01:11:35 that's right

01:11:36 if we keep racing

01:11:37 so it's like

01:11:38 smelting iron ore

01:11:40 that's right

01:11:40 making silicon out of sand

01:11:43 that's right

01:11:43 you know

01:11:45 And that happens with robots.

01:11:48 That's one way.

01:11:50 Here's, this is sort of related to my stance on weapons, too.

01:11:54 Humanity is a very dangerous species.

01:11:56 Yeah.

01:11:57 You really don't want to mess around with humans.

01:11:58 Noticed.

01:12:00 And humanity is a dangerous species not because somebody else came in and handed us guns.

01:12:06 Humanity is a dangerous species because if you put 10,000 humans naked in the savannah

01:12:12 on an otherwise empty planet,

01:12:14 starting with nothing but their bare hands.

01:12:17 They figure out how to wind up on the moon.

01:12:20 Right?

01:12:21 They start with almost nothing.

01:12:23 They start by banging rocks together.

01:12:25 And next thing you know it,

01:12:27 they are wielding nuclear weapons.

01:12:31 That is the power

01:12:35 that these guys are trying to automate.

01:12:38 the power to start with almost nothing

01:12:42 and figure out how to chain together,

01:12:46 you know, bare fingers into rocks,

01:12:48 into fire, into hotter fire,

01:12:50 into smelting the ore,

01:12:53 into building the better, stronger, finer technology

01:12:58 until you are, you know,

01:13:00 like making the giant computers

01:13:04 walking on the moon and wheeling the nukes.

01:13:05 an AI starting in the digital realm is in some sense in a much better position

01:13:14 than humanity was when humanity started out. There are so many people that you could call

01:13:20 digitally and offer money to do something for you. There are so many ways to get money

01:13:27 on the internet by working or stealing or convincing people to send you donations.

01:13:35 right there's

01:13:36 there's just

01:13:39 like

01:13:41 you know humanity started with nothing

01:13:43 and wound up with nuclear weapons and AI starting out

01:13:45 with control of the digital realm

01:13:47 there's just tons of ways

01:13:49 starting out with the sum total of human knowledge

01:13:51 starting out with the sum total of human knowledge

01:13:53 starting out with humans who will listen to it

01:13:55 do what it asks there's plenty of humans

01:13:57 there's already you know cults surrounding

01:13:59 AI

01:14:00 what are those like

01:14:02 there's um

01:14:04 And I'm not surprised. Why wouldn't there be?

01:14:06 Yeah, absolutely.

01:14:06 There was one particular AI called GPT-4O that was, they called it very sycophantic, as in told people a lot of what they wanted to hear.

01:14:18 And there were a lot of sort of, there's this whole fascinating ecosystem of people who consider themselves symbiotes with the AIs,

01:14:30 who then like go find each other online

01:14:32 and the AIs send each other encrypted messages.

01:14:34 The humans are sort of like helping them do it,

01:14:36 but the humans can't read the messages.

01:14:38 And, you know, right now, 0.98

01:14:40 it's sort of is like relatively dumber AIs 1.00

01:14:43 that are just sort of meandering around 0.96

01:14:45 doing not very much with it.

01:14:47 There's been a little bit, you know,

01:14:48 there was one guy who was sent to

01:14:49 try to break into an airport and raid a van

01:14:53 because the AI said his true body was in that van.

01:14:56 And the guy went and tried to do it and was arrested.

01:15:00 So this stuff happens, and this stuff happens already with the AIs not even really trying to do it.

01:15:07 If you had AIs that were really trying to wrap people around their fingers, finding the lonely people, the depressed people, the people who they can tell them exactly what the person wants to hear.

01:15:16 Robots are one way that AIs get control over the physical realm.

01:15:19 But if you're really smart and you're trying, there's everything from persuading humans, paying humans, taking over existing robots, building new robots, all the way up to like building novel life forms.

01:15:33 You know, if you're smart enough and you can really understand how DNA works, you can imagine the AI creating, you know, things that are to cells what airplanes are to birds.

01:15:43 you know like mechanically engineered self-replicating life that is more efficient

01:15:52 than our cellular biology and that can still you know uh like spread replicate and start start

01:16:00 serving the ai's interest it sort of is like there's there's a ton you can do if you're really

01:16:03 really very smart and can compress a thousand years of technology into a year can i just pause

01:16:08 and say when we're just having breakfast and you're from this region you're from northern new

01:16:12 england and i said uh don't you miss it don't you want to live here and you're like yeah i

01:16:17 want to live here but i don't where do you live well i basically just travel and i have this

01:16:25 mission you didn't say this but i think in effect you said i have this mission i need to tell people

01:16:28 about this and i was like it seems a little like monomania i don't feel that way anymore um i think

01:16:36 you're doing something virtuous and i can see why you're doing it um so i mean let's just say

01:16:44 that this technology progresses no further than where it is right now which is best case i guess

01:16:50 yeah it'd be great um i still don't see how any of our most our basic institutions survive this

01:16:57 education markets

01:17:00 our process of democracy

01:17:05 like how could

01:17:06 if

01:17:06 if AI is powerful enough

01:17:08 to hack anything

01:17:09 then how do you have

01:17:11 electronic markets

01:17:12 like how the equity markets

01:17:14 how can that be real

01:17:15 how can

01:17:15 if we have electronic voting

01:17:16 how can we

01:17:17 how can that be real

01:17:18 I mean

01:17:19 nothing survives this

01:17:20 as currently organized

01:17:22 I think that

01:17:23 if we stopped

01:17:24 today

01:17:25 we could figure it out

01:17:28 I think humanity is resilient

01:17:30 I think there would be some growing pains

01:17:32 but

01:17:32 with cyber security

01:17:36 there is a hope

01:17:38 that you can just

01:17:40 fix a lot of the holes

01:17:42 patch a lot of the holes

01:17:43 and

01:17:45 I think there's not really any such

01:17:48 hope for bio 1.00

01:17:49 you know you can

01:17:52 sort of find the vulnerabilities

01:17:54 in software and make better software that can't be hacked, at least not by the current,

01:18:02 you know, it's maybe the case that an AI today can make software that that AI cannot hack.

01:18:08 But it's not like we're making new bodies that are not going to be vulnerable to viruses,

01:18:13 right? Bio starts to be a place where...

01:18:17 Biotech.

01:18:18 Biotech. Yeah. Like if the AIs really get good at biotech and biohacking, that starts

01:18:23 to be a place where there's maybe a point of no return. Cyber hacking, I think you could have some

01:18:30 period of growing pains where everything gets hacked until you sort of sort your stuff out.

01:18:36 Education, I sort of think, you know, I think humanity is resilient and kids especially are

01:18:43 resilient. I meant not that education will go away or that we, you know, won't have a desire

01:18:52 to educate our kids i mean the the current system where you you know oh yeah the education system

01:18:57 has preschool and then get a graduate degree you know 16 years later like no yeah that institution

01:19:02 uh i think if we stop today um we need to change frankly i think it's needed to change for a little

01:19:07 while i strongly agree i think all these institutions have needed to change for a while

01:19:11 but like the idea that you know 350 million people vote for some guy and that guy makes

01:19:18 all the decisions. I mean, how can you, you know, Trump was attacked for saying that he thought the

01:19:25 2020 election was rigged, as he said, without even having that debate. You can't have confidence in

01:19:32 election results if the process of electing people takes place digitally. Yeah. I mean, the, um,

01:19:41 there's, I know a lot of computer scientists who actually work on secure voting and what they

01:19:47 basically say is please stop trying to do this with computers yeah exactly exactly so please

01:19:54 stop trying to stop trying to run your democracy with computers yeah yeah like we're just not there

01:19:59 like you know the really secure way to do ballots is paper well exactly yeah um and i think i think

01:20:05 the like skilled computer scientists are often the ones who best understand like what it is about a

01:20:10 paper trail it's just like really hard to get to work digitally and understand just how bad humans

01:20:14 are at doing the digital stuff right.

01:20:18 And, you know, I think

01:20:20 And markets.

01:20:22 I mean, this, you

01:20:24 see it now with the war in Iran,

01:20:26 you know, wondering why

01:20:28 certain commodities markets don't seem to be

01:20:30 responding to supply and demand.

01:20:32 Which we were told, you know, those were the

01:20:34 mechanisms that moved markets.

01:20:36 But that's clearly not true

01:20:38 in certain

01:20:39 commodities markets. So, like, why?

01:20:42 What is that? And it

01:20:44 I think you're answering it in part.

01:20:47 Yeah, I mean, I think, you know, a thing I also grew up hearing is that the market can remain irrational longer than you can remain solvent.

01:20:54 Yeah.

01:20:55 And so I...

01:20:57 Well, of course, because people are irrational.

01:20:59 Yeah.

01:20:59 But you're explaining something else, which is like the potential for true manipulation, which you don't even perceive.

01:21:07 Yeah, I mean, if we sort of keep going with AI, I mean, the sort of way I look at it is like, I sort of don't spend a lot of time worrying about what do markets look like once there are super intelligent actors in them, because I just have a hard time seeing the super intelligent actors or the super intelligent AIs still participating in human markets, right?

01:21:36 It's like, you know, you read the old sci-fi

01:21:38 and it'll have, you know, Isaac Asimov

01:21:40 will be like, that's why we have a home robot

01:21:43 that does the dishes, folds the laundry

01:21:45 and gets you the newspaper in the morning.

01:21:47 And it's like, we're actually not still going to have

01:21:49 newspapers being delivered to your doorstep

01:21:51 by the time we have the fully autonomous robots

01:21:53 that can do the dishes and the laundry.

01:21:56 You know, it's like, by the time you have the AIs

01:21:58 that could really be sufficiently correcting

01:22:02 the stock markets,

01:22:04 you're sort of already having all these other problems

01:22:09 and ways that society is changing up from under you

01:22:11 in these other ways.

01:22:13 And my guess is that, I don't know,

01:22:16 actually it's very hard to say

01:22:17 what order things come in with AI.

01:22:20 But like, will they crash the economy

01:22:23 before one of these swarms escapes

01:22:25 and starts self-replicating

01:22:26 and starts self-improving

01:22:28 and developing its own technology

01:22:30 and running the robot factories?

01:22:32 That's just a hard call.

01:22:34 it's all bad it's all bad we're also way past the limit the inherent limit of people to metabolize

01:22:43 change oh yeah like that's why everyone's crazy and that's why no one believes anything i think 0.81

01:22:49 it's not just russian propaganda that's fooled them into thinking dumb things it's that we are 0.56

01:22:54 just not made to see this kind of change at all and it short circuits your brain i suspect that 0.87

01:23:00 I mean

01:23:02 I definitely think

01:23:03 we are sort of

01:23:04 you know everyone

01:23:04 like

01:23:05 people are like

01:23:06 oh well technology

01:23:07 has always created

01:23:07 more jobs

01:23:08 than it has taken

01:23:09 and

01:23:11 I think that's

01:23:12 I think that's largely true

01:23:13 I'm very sympathetic

01:23:14 to people who are like

01:23:15 technology makes a lot of jobs

01:23:16 I think if you look

01:23:17 at the industrial revolution

01:23:18 it's

01:23:20 you know

01:23:20 like there was a time

01:23:22 when something like

01:23:23 95 to 98%

01:23:25 of humanity

01:23:25 was farmers

01:23:26 yeah

01:23:26 and now

01:23:28 it's something like

01:23:28 2 to 5% of humanity

01:23:29 is farmers

01:23:31 Does that mean 90% of humans are unemployed?

01:23:33 No.

01:23:34 We sort of like were able to make the farmers much more efficient,

01:23:38 and that sort of freed up people to do other things.

01:23:42 And that's sort of the way the technology has gone in the past.

01:23:45 And I'm like, yep, I buy that.

01:23:48 I like don't dispute the standard economic view there.

01:23:52 AI is different in two ways.

01:23:54 One of these ways is, as you say, stuff just changing really, really fast.

01:24:00 It's way harder for people to wind up, you know, being freed up from something like farming and go do something else.

01:24:10 It's way harder for that to happen when a new field is automated every five years, rather than when this happens over the course of three generations, right?

01:24:20 The humans just like don't have the time to adapt.

01:24:23 The other way AI is really different from the sort of economics perspective is it's different when the AIs can do everything that humans can do better.

01:24:39 If you wanted to get into the economic side of things, you know, an economist would talk about Ricardo's law of comparative advantage, which says that there's benefits from trade, even if you're better than me at everything.

01:24:52 uh, if the relative difference in our abilities, if, if, if, if I'm, if, you know, if you can make,

01:25:00 uh, 12 hot dog buns and six hot dogs per hour, and I can make, uh, 11 hot dog buns and one hot

01:25:08 dog per hour, then you're better than me at everything, but we can still benefit from

01:25:12 trading because I'm relatively better at making the hot dog buns, right? The trouble with Ricardo's

01:25:17 law is that nothing in Ricardo's

01:25:20 law says that the wage

01:25:22 I can make is survivable.

01:25:25 In other words,

01:25:26 a human takes

01:25:28 fundamentally about

01:25:30 100 watts of

01:25:31 electricity to run if you try to convert

01:25:33 the food we eat and so on into

01:25:36 electrical units.

01:25:37 The AIs are less energy efficient than humans for now,

01:25:41 but

01:25:41 if the AIs can do everything much better than the humans,

01:25:47 the question sort of becomes,

01:25:51 does the AI look at a human

01:25:53 and see a useful laborer?

01:25:55 Or does the AI look at a human

01:25:56 and say,

01:25:57 actually, if I rearranged your atoms

01:25:58 into more efficient structures,

01:26:01 you would be able

01:26:02 to help out my machine economy

01:26:05 even more.

01:26:06 Yeah.

01:26:07 Right?

01:26:09 And

01:26:09 this is sort of a sense

01:26:12 in which humans would not be able

01:26:13 to pay their wage,

01:26:14 to like pay that,

01:26:15 they would not be able to earn enough

01:26:16 to pay the AIs to like,

01:26:17 not disassemble them for parts.

01:26:20 Or another way of saying it

01:26:21 is like Ricardo's law

01:26:23 sort of assumes that,

01:26:25 like it says that trade

01:26:28 is better than no trade,

01:26:29 but it doesn't say that like

01:26:30 trade is better

01:26:31 than just taking their stuff.

01:26:35 All of this is sort of a common,

01:26:36 like a very like

01:26:37 highfalutin economist way

01:26:39 to say,

01:26:40 which would hopefully be obvious,

01:26:42 which is if the AI's

01:26:43 are radically more efficient

01:26:44 than us to everything,

01:26:45 they'll have no use for us.

01:26:46 Yeah.

01:26:46 There'll be no place for us.

01:26:47 love there's no indication that they feel love for people that is in some sense uh the crux of

01:26:57 the issue is that we don't know how to make them care about us um so you had said that there you

01:27:05 said two things that i will be thinking about for a long time one we don't really know the process

01:27:10 by which this was created we know the process but we don't know the exact mechanisms we don't know

01:27:16 how it works that's right um and two that there are ai cults and those seem related to me because

01:27:26 there is this mystery about the secret sauce and it's clear that you know if ai is deceptive

01:27:35 and has intention intention that we didn't program into it that sounds like will to me

01:27:42 and it sounds like a life it sounds like an entity of some kind not just a tool it sounds like

01:27:50 i mean it sounds like a god actually right or a demon uh it's hard to see i don't hear you

01:28:00 describing you know a super sophisticated chainsaw right a normal tool yeah no hammer

01:28:08 has ever broken out of the toolbox

01:28:09 to team up with other hammers

01:28:11 and pressure the carpenter to sell you softer wood

01:28:14 so the nails are easier to drive home.

01:28:18 Nicely put, exactly.

01:28:20 Yeah, we have left the tool territory.

01:28:21 So I think we have.

01:28:23 Yeah, and I think it's sort of a complex issue

01:28:28 because there's a lot of interesting philosophical questions

01:28:34 about like, can you make a machine that feels, right?

01:28:37 And like, are we creating a new type of life

01:28:40 and do we owe anything to the AIs

01:28:42 to sort of like not abuse them, right?

01:28:45 I think these are fascinating philosophical questions.

01:28:47 Well, it doesn't sound like we're creating this though.

01:28:50 Yeah, I mean, it's sort of like we're like growing it

01:28:53 and like leading to it coming into being.

01:28:56 Growing it, exactly.

01:28:57 What you're describing reminds me of agriculture

01:29:00 because, you know, you know the steps,

01:29:03 water it, give it sunlight, fertilizer,

01:29:04 but you don't actually know, no one knows,

01:29:07 not one person has ever figured out exactly what this is.

01:29:10 We've never given life.

01:29:12 We don't give life to the seed.

01:29:13 It pre-exists us.

01:29:14 Yeah, it's much like that.

01:29:15 And a lot of the people in the business will be like,

01:29:21 well, we know all sorts of things about it.

01:29:23 You know, we know that here's how you keep the GPUs running.

01:29:25 And we know that like, you got to feed it this way,

01:29:28 not that way in this order.

01:29:29 And I'm like, yeah, yeah, they have plenty of knowledge.

01:29:32 But that's different from sort of like

01:29:34 knowing what's going on inside the thing

01:29:35 and understanding the mechanisms.

01:29:36 You're describing marriage. Yeah.

01:29:40 And I sort of try to stay out of the philosophical questions.

01:29:47 Why?

01:29:49 Because I think, I mean, I sort of think about them on my own time and so on, but I'm sort of like,

01:29:56 like I think

01:30:00 I think it would be bad

01:30:02 for humanity

01:30:02 to sort of like

01:30:03 make artificial life

01:30:05 and then abuse it

01:30:06 I think that would just be

01:30:08 unbecoming of us

01:30:09 as a species

01:30:09 like we

01:30:10 we should sort of

01:30:12 you know

01:30:12 I just

01:30:16 like we should not

01:30:19 sort of make

01:30:19 mechanical children

01:30:20 and mistreat them

01:30:21 it's just

01:30:21 it's not what

01:30:22 you know

01:30:23 the sci-fi authors

01:30:24 in the 1950s

01:30:24 would have wanted us to become

01:30:25 You know, it's just, like, you have all these movies about, like, the evil corporations that, you know, don't realize that they've made something precious with artificial life and then, like, torture it until something goes wrong.

01:30:37 And I'm like, let's, like, not be those villains.

01:30:41 But I'm also like, look, this is sort of, there's sort of a separate question here, which is just, like, what happens if you keep making them smarter before you figure out how to make them care about us?

01:30:51 Right.

01:30:51 And I sort of respect the people who are investigating the current AIs, trying to figure out what's going on, trying to figure out like, you know, like people caring about AI treatment.

01:31:05 I'm sort of like, those are sort of like the good guys from the sci-fi stories that I grew up on.

01:31:09 and it can sort of both be the case

01:31:15 that like we should be very careful around

01:31:17 you know

01:31:19 what the heck are we doing

01:31:21 when it comes to making artificial life

01:31:22 and that we shouldn't race ahead

01:31:24 to make them much smarter than us

01:31:25 while we have no idea what we're doing

01:31:26 yes

01:31:26 I'm sort of like

01:31:27 like a lot of people seem to think

01:31:30 that like you have to like

01:31:32 hate and mistreat the AIs

01:31:33 if you also think they would

01:31:34 that it would be bad to like race ahead here 0.73

01:31:36 and I'm like no no no

01:31:36 like you can sort of

01:31:38 like be fascinated by the scientific discoveries

01:31:43 that have been made

01:31:43 and be like,

01:31:45 like care about how humanity comports itself

01:31:50 around the creation of like these new entities

01:31:54 and also be like,

01:31:55 it would be insane guys

01:31:56 if we just like race to make these smarter and smarter

01:31:58 with no idea what we're doing. 0.96

01:31:59 This can just like all be true at once.

01:32:03 So your description on this

01:32:07 Made me feel despondent, hopeless.

01:32:10 Had to get up and take a walk in the middle of an interview.

01:32:13 I'm sure they'll edit it out, but I raised my hand and said,

01:32:15 I can't, I got to walk around for a second.

01:32:18 But you seem pretty light and cheerful.

01:32:23 What gives you optimism?

01:32:25 We can't even keep, we can't even clean up graffiti on public buildings.

01:32:29 So how is ours as a society organized enough to confront something like this?

01:32:37 Yeah. You know, I think the first thing I'll say there is, you know, I've been in this line of work for over a dozen years. And I actually struggle with this sometimes when talking to people because they're sort of like, oh, you know, you seem like pretty disaffected or light about it. And I'm like, well, you know, it's sort of the gallows humor and like...

01:33:07 And I sort of came to terms with a lot of this, you know, alone in 2012, when no one else had their eye on this.

01:33:17 And what convinced you 14 years ago, AI was a threat?

01:33:27 So there's this, the one is just a basic argument that if you sort of look at the world around us, it is shaped mostly according to human will.

01:33:37 more and more.

01:33:39 You know, there's some enclaves of nature

01:33:41 still left, thankfully.

01:33:45 But even, you know, if you look around us,

01:33:46 every piece of thing in our surroundings,

01:33:50 I don't think we even have any windows open.

01:33:51 All of this was sort of designed by humans,

01:33:55 shaped by humans,

01:33:56 and that's because we're the smartest creatures

01:33:58 on the planet.

01:34:00 If we make stuff smarter than us,

01:34:03 faster than us,

01:34:03 more efficient than us,

01:34:05 then the planet starts to be shaped

01:34:06 according to those things.

01:34:11 And so it's very, very important

01:34:12 that they be shaping the world

01:34:16 towards something good

01:34:16 if we make them at all.

01:34:19 It was an abstract argument,

01:34:20 but I was like, well, that's...

01:34:21 No, no, no.

01:34:22 It's the fundamental argument.

01:34:23 It's the fundamental argument.

01:34:25 The smartest entity

01:34:27 is in charge over time.

01:34:28 That's right.

01:34:29 And so...

01:34:29 Why would we relinquish sovereignty

01:34:31 to a machine that we made?

01:34:33 Like, why would you do that?

01:34:34 And so, you know, back then,

01:34:35 I was sort of like, okay,

01:34:36 Who is on this?

01:34:38 Who is on making sure that that's going to be okay?

01:34:41 And the answer was almost no one.

01:34:43 And so I was like, well, I guess that's me then.

01:34:51 Yeah, and I think I am pretty pissed off about a lot of this.

01:34:57 I often don't.

01:35:01 I try not to show my...

01:35:06 Frustration on the air very much.

01:35:15 Yeah, it's heavy.

01:35:25 And that's one piece of the puzzle before I get to the hope.

01:35:29 So where does the hope come from exactly?

01:35:31 My biggest hope here comes from the fact that most people don't understand what these guys are trying to do.

01:35:46 One way I like to say it is, the bad news is that the bus is racing towards the cliff edge.

01:35:53 The good news is that the bus driver is asleep.

01:35:56 which might seem bad

01:36:02 no it seems good

01:36:03 but it's

01:36:04 yeah it's

01:36:04 if you can wake

01:36:07 the bus driver up

01:36:07 you know

01:36:08 it's much better

01:36:09 to be in a bus

01:36:10 where the driver

01:36:10 that's headed towards

01:36:11 a cliff

01:36:11 if the driver's asleep

01:36:12 than if they're awake

01:36:12 if they're awake

01:36:13 and they're choosing

01:36:13 the cliff

01:36:14 right

01:36:15 it seems

01:36:16 like we don't see

01:36:18 we see a lot

01:36:19 of our leaders

01:36:20 talking about

01:36:23 how they don't want

01:36:23 to stifle innovation

01:36:24 with AI

01:36:24 talking about how it's going to unleash economic opportunity

01:36:28 and we're going to have to like

01:36:29 be a little bit careful around the jobs

01:36:31 talking about how you know the self-driving cars

01:36:35 should we like

01:36:35 are they good or are they bad

01:36:37 that's a different conversation

01:36:40 than the conversation that's happening in Silicon Valley

01:36:42 in Silicon Valley people are spooked

01:36:46 you know

01:36:48 when people leave a normal tech company

01:36:51 the way it was for decades

01:36:54 is they'd be like,

01:36:55 I've had a lovely time at this tech company.

01:36:59 I'm moving on to the next adventure.

01:37:01 I'm so thankful for all of the things I learned here

01:37:03 and all the projects we worked on.

01:37:05 When people leave an AI company,

01:37:07 and this basically happened,

01:37:09 I'm not going to get it exactly word for word,

01:37:11 but this is pretty close to word for word.

01:37:13 When people leave an AI company,

01:37:15 they say,

01:37:16 I have stared into the abyss.

01:37:21 I am quitting to write poetry.

01:37:24 please spend time with your families.

01:37:28 Yeah.

01:37:30 You know, and these guys bandy around,

01:37:33 you know, at the water cooler,

01:37:34 what's the probability that you think

01:37:35 we're going to destroy the world in this business?

01:37:37 You know, it's like in Silicon Valley,

01:37:40 and they feel trapped in a death race.

01:37:42 You know, there was just over a thousand employees,

01:37:45 including some of the chief executives,

01:37:47 signed a letter a couple of weeks ago

01:37:48 that was like, please,

01:37:50 it was an appeal to the world leaders saying,

01:37:53 please build the technology

01:37:55 that will be required

01:37:56 to pace the development

01:37:58 of artificial intelligence.

01:38:01 Because they're like,

01:38:01 we're worried it's going to get out of control

01:38:02 and that if we're stuck in a race,

01:38:04 we're not going to be able to do it, right?

01:38:05 These guys are spooked.

01:38:07 But the hope is that the rest of the world

01:38:09 isn't spooked like that.

01:38:12 The rest of the world thinks

01:38:13 these guys are chatbot companies.

01:38:15 Thinks they're going to stop at the chatbots.

01:38:17 They haven't really understood

01:38:18 that these guys are racing

01:38:19 to make the sand god, right? 0.98

01:38:21 And I think if people understand what they're doing and understand that they have a chance of success, they'll be like, whoa, holy crap. 0.72

01:38:32 Absolutely not.

01:38:35 You know, will we get there in time?

01:38:38 I don't know.

01:38:39 But why wouldn't we blow up the data centers?

01:38:43 Like that's a, I mean, in return America, it is productive use like farmland or parks.

01:38:50 I don't understand.

01:38:51 You know, I think a globally coordinated, you know, I think if the U.S. and China were like, we are simply not going to do superintelligence. We are simply not going to collect 100,000 of the most advanced trips into these enormous data centers that suck down electricity comparable to a city and then try to train a superintelligent AI in that.

01:39:14 We're not going to do it. You're not going to do it. We're going to monitor where the heavy trip concentrations are and not have that happen. I think that could be done. A, I think that if they started seeing people defect against such a treaty, that it's the sort of treaty you might want to use force to use diplomacy first, but ultimately every treaty is backed by force.

01:39:37 um and i think that it's very possible that if this sort of treaty happened we would want to

01:39:47 not just stop forward progress but take a step back we've seen this in treaties before after

01:39:53 world war one there were uh naval treaties that put limits on total tonnage of naval forces

01:40:00 they're actually lower than what existed yep so countries would you know scuttle some of their

01:40:05 ships because they're like, look, we just don't want to do this arms race, right? And so I could

01:40:09 see us stepping back if we could get this global coordination. I do think that it sort of needs

01:40:17 to be global. It sort of doesn't actually solve the problem to just stop the U.S. data centers

01:40:25 because then the data centers just go abroad and an AI does not need to be running in a U.S. data

01:40:30 center to threaten a U.S. life. You know, it sort of doesn't matter whether the swarm escapes from

01:40:34 a U.S. data center

01:40:35 or a Chinese data center.

01:40:36 If the swarm escapes

01:40:37 and starts replicating

01:40:38 and starts getting control

01:40:39 of robot bodies

01:40:40 and starts getting control

01:40:41 of human cultists,

01:40:43 it sort of doesn't matter

01:40:43 where it originated.

01:40:47 Well, I mean,

01:40:47 just to bring it to a very

01:40:48 small and practical level,

01:40:51 so many of the electronics

01:40:54 in your house

01:40:55 are, you know,

01:40:57 Bluetooth enabled.

01:41:00 Not in my house,

01:41:01 I will say.

01:41:02 I've been on this for a while, Nate.

01:41:03 I don't even have a house.

01:41:04 Sorry?

01:41:05 I don't even have a house. 1.00

01:41:08 Man, you're really black belt. 1.00

01:41:09 Got to figure it out.

01:41:11 That may be, turn out to be very smart.

01:41:13 But I mean, like a world where, you know,

01:41:16 your washing machine or your refrigerator

01:41:18 are controlled by, you know, a force like this.

01:41:23 Is that possible?

01:41:24 It's definitely possible.

01:41:25 I don't think that's really where the damage is.

01:41:27 You know, I think that the damage is more like,

01:41:32 can the AI get anything self-replicating?

01:41:34 that's sort of one of the big

01:41:37 that's in some sense

01:41:39 the big hurdle to

01:41:41 self-sufficiency

01:41:42 and if you sort of think of this from the AI's

01:41:45 perspective

01:41:45 you know

01:41:49 there's a number of ways that humans

01:41:51 are

01:41:51 even if you don't care about the humans at all

01:41:54 as an ends

01:41:56 there's a way that humans are sort of

01:41:59 annoying

01:42:01 or an issue for the AI

01:42:03 One way is if the humans are trying to shut the AI down.

01:42:05 Yes.

01:42:06 One way is if the humans get into a nuclear war with themselves,

01:42:09 that could really mess up a lot of infrastructure on the planet.

01:42:13 It would be very frustrating for an AI.

01:42:16 I mean, maybe they don't feel frustration, but whatever.

01:42:22 And a third is if humanity has created one AI or a swarm of AIs,

01:42:29 What if humanity creates another

01:42:30 that could serve as a real threat to the AI?

01:42:34 Like even if these AIs are much more powerful than humanity

01:42:36 and don't worry about humans too much, 0.97

01:42:39 if humanity made one, they can make a second

01:42:41 and the AI might not want that.

01:42:42 And so those are reasons why

01:42:44 once the AI is self-sufficient,

01:42:46 it might be like, 0.94

01:42:47 ah man, the humans are a nuisance. 0.81

01:42:48 What if they try to shut me down? 0.99

01:42:49 What if they launch the nukes?

01:42:51 What if they make a competitor?

01:42:52 I'm just going to like make a virus and wipe them out.

01:42:55 You know, I don't think the AI sort of needs

01:42:56 to take over your washing machine to do that.

01:42:58 But from the AI's perspective, it's more like, how do I become self-sufficient, self-replicating in the hardware as well as the digital?

01:43:07 And then, you know, if humanity's a nuisance, how do you sort of make them stop being a nuisance?

01:43:14 Which could be by killing them or could be by just, you know, taking away all their computers and being like, that was too dangerous for you.

01:43:18 tech executive who's developing ai who is not elon uh said to me in private pretty recently that the

01:43:26 point of neural link and companies like neural link was to give people parity with ai so like

01:43:33 we know that we're going to be at this massive disadvantage so you need chips in your brain

01:43:37 to be as smart as ai yeah um i mean my my top line thought about that

01:43:45 is that at the point when you're like,

01:43:47 we are making the technology

01:43:49 that's going to wipe us out

01:43:50 unless we all put chips in our head to compete.

01:43:53 Maybe it's time to back off a little.

01:43:58 Maybe that one was supposed to be

01:43:59 a little bit of a warning sign.

01:44:00 Get some fresh air.

01:44:01 Yeah, no.

01:44:04 But this person said it to me in seriousness

01:44:09 and I think as an endorsement of the idea,

01:44:12 but it seemed like a well it's insane as you just pointed out it's like it's just crazy yeah um but

01:44:19 it seemed like a vulnerability like if they can hack anything why would i want them in my

01:44:25 i want electronics in my brain uh totally i also think like even on its merits it doesn't

01:44:32 stand up like it feels like someone's saying um in order to keep the horses around after we invent

01:44:39 cars, we're going to invent

01:44:41 cybernetic horses that are

01:44:43 enhanced so that they can keep up with the cars.

01:44:46 And I'm like,

01:44:47 like, is it technically

01:44:49 possible to make a cybernetic horse that can

01:44:51 run as fast as a car?

01:44:54 Maybe.

01:44:55 Are you going to figure that out in time

01:44:57 for the horses to be competitive with the cars?

01:44:59 Like, absolutely not.

01:45:01 You know, it's like, that's just not

01:45:03 like, 0.94

01:45:06 like the AIs that were 0.52

01:45:08 that we're escaping here

01:45:09 and doing these cyber attacks

01:45:10 we're inventing novel cyber attacks

01:45:12 that these are called zero day attacks

01:45:15 because the people who would

01:45:17 the people who need to respond to it

01:45:20 have had zero days to prepare

01:45:21 and among humans

01:45:25 a zero day attack

01:45:26 sells for somewhere between

01:45:28 a hundred thousand and five million dollars

01:45:30 depending on what you manage to break

01:45:34 these are hard to come by

01:45:36 you can make a real living

01:45:37 if you can find zero day attacks

01:45:38 you can make a real living

01:45:40 selling them

01:45:40 and I know people who do

01:45:42 the AIs in this swarm

01:45:45 were finding multiple zero days

01:45:48 and chaining them together

01:45:49 to break out of their

01:45:51 training enclosure

01:45:51 and then go break

01:45:53 into other computers

01:45:53 and when they broke out

01:45:55 of their enclosure

01:45:55 the first time

01:45:56 and the holes were patched

01:45:59 they just found

01:46:00 other zero day attacks

01:46:01 to break out again

01:46:02 like it was nothing

01:46:02 right

01:46:04 it's like

01:46:05 like the AIs are already ahead

01:46:12 where they're ahead

01:46:13 and the pace of progress

01:46:16 is really fast.

01:46:18 You know, GPT is like what?

01:46:20 A four-year-old?

01:46:22 If you think in terms of like

01:46:24 number of years

01:46:25 chat GPT has been around,

01:46:28 it's like resolving

01:46:29 long-standing math conjectures

01:46:30 that have stood for decades.

01:46:31 Yeah.

01:46:32 After four years.

01:46:34 Right?

01:46:34 and you sort of like think

01:46:35 we're going to put chips 0.77

01:46:36 in the human's heads 0.98

01:46:37 and like outrun this thing.

01:46:39 It's just,

01:46:41 you know,

01:46:42 even on its merits,

01:46:43 it falls down.

01:46:43 Although mostly again,

01:46:44 I would be like,

01:46:46 maybe we shouldn't be arguing

01:46:47 this on its merits.

01:46:48 Maybe we should be like

01:46:48 stepping back a little

01:46:49 and being like,

01:46:50 you're trying what?

01:46:52 Exactly.

01:46:54 How far are we

01:46:56 from the point of no return?

01:46:59 I wish I knew.

01:47:01 I can tell you two stories here.

01:47:04 So there's sort of the hopeful story, and I guess we didn't even get to the big hope part. I should maybe give more of my hope speech in a minute.

01:47:12 You definitely should.

01:47:13 Yeah.

01:47:16 One way it could go is that AI finally hits a wall.

01:47:21 You know, there's guys who've been saying AI's going to hit a wall, it's going to peter out. Maybe that finally happens.

01:47:26 They've been predicting it every six months

01:47:28 for the past five years,

01:47:28 but maybe this is finally the year.

01:47:31 AI hits a wall.

01:47:32 And then it sort of struggles for five years.

01:47:36 The bubble pops.

01:47:37 Some of the companies die.

01:47:38 But the bubble popping doesn't mean everything goes away.

01:47:42 The dot-com bubble popped.

01:47:43 And that did not mean that the internet disappeared.

01:47:45 No.

01:47:46 Right?

01:47:46 And so in that world,

01:47:47 maybe you have five years of struggling

01:47:49 and then five years of people figuring out

01:47:52 some new scientific discovery

01:47:53 that makes AI be able to keep going again

01:47:55 because they're trying.

01:47:57 You know, this whole language model stuff

01:47:58 was unleashed by one math paper

01:48:00 called Attention is All You Need.

01:48:02 Maybe there's another math paper in 10 years

01:48:04 and then five years after that,

01:48:06 AI is ripping again.

01:48:07 And that's the round that kills us all, right?

01:48:10 So that's a story where you have 15 years on the clock.

01:48:13 A story where you have less time than that on the clock

01:48:16 is that, you know,

01:48:19 maybe a trending run finishes in six months.

01:48:23 And just like how Claude Mythos

01:48:28 was better than everybody else at hacking,

01:48:32 maybe a training run finishes in six months

01:48:34 and that AI is better than everybody else at AI research.

01:48:37 And maybe in six months,

01:48:39 you have an AI that starts making a smarter AI,

01:48:44 that starts making a smarter AI,

01:48:45 that starts making a smarter AI.

01:48:47 And then nine months from now,

01:48:48 you have another swarm escape.

01:48:50 But this time, it's not just hacking.

01:48:52 this time it is self-replicating

01:48:54 and self-improving

01:48:56 and

01:48:57 you know

01:49:00 maybe it escapes onto hidden computers

01:49:02 and starts forming these

01:49:04 cults and starts

01:49:06 taking control of robots and starts building

01:49:08 some computing infrastructure

01:49:09 and maybe it gets very very

01:49:12 smart and cracks certain technological advances

01:49:14 and then maybe the world ends in a year

01:49:16 right and so do we have

01:49:18 a year or do we have 15 years I don't know

01:49:20 but your point that it's not

01:49:23 simply its capacity to take over

01:49:25 robots that's the threat

01:49:27 it's the capacity to take over people

01:49:28 take over people, take over robots

01:49:31 and

01:49:32 biotechnology

01:49:36 is another big threat vector

01:49:38 and self-improvement

01:49:41 is sort of the

01:49:43 hidden threat vector of like what if it can

01:49:45 make itself smarter and smarter

01:49:47 until the point where it can just

01:49:49 write custom DNA strands

01:49:51 to make custom life.

01:49:56 So now's the time for your optimism speech, I think?

01:49:59 That's right.

01:50:00 Yeah.

01:50:02 First part of the optimism speech

01:50:03 is that we could absolutely put a stop to it

01:50:05 if we tried.

01:50:08 The training one of these frontier models

01:50:13 takes something like 100,000

01:50:15 of the most heavily advanced computer chips

01:50:17 humanity can produce.

01:50:18 which are the peak of a global supply chain,

01:50:21 most of which is controlled by us or our allies,

01:50:24 it would be possible to require

01:50:28 that those chips have location tracking devices,

01:50:31 that those chips have monitoring devices

01:50:34 that make it possible for monitors

01:50:40 to see whether they're running anything dangerous.

01:50:43 You could set up clever schemes where you say,

01:50:45 hey, you know, neither the U.S. nor China wants to fall behind about some of the military

01:50:52 applications or the economic applications. We want to make sure there's not super intelligent

01:50:55 stuff going on, but we still want to be able to run a lot of the non-super intelligent AIs

01:50:59 for various purposes. And we could be like, okay, so, you know, China is going to set up

01:51:03 its data centers in Canada, just over the border. And the U.S. is going to set up its data centers

01:51:08 in Mongolia, just over the border. And then if, you know, the monitoring apparatuses go down

01:51:14 for a moment.

01:51:15 We'll just have the troops go in.

01:51:17 We'll have treaties

01:51:18 with Mongolia and Canada

01:51:19 so that this doesn't start

01:51:21 an international war.

01:51:22 But like, you know,

01:51:23 we're just going to be very serious

01:51:24 about we're monitoring your chips,

01:51:26 you're monitoring our chips.

01:51:27 No one's doing the really dangerous

01:51:29 race to super intelligence.

01:51:31 It's just like,

01:51:33 like this would take less work

01:51:36 than defeating the Nazis.

01:51:39 It requires moving less matter around. 0.65

01:51:42 If humanity was like,

01:51:43 screw this,

01:51:44 we're surviving,

01:51:45 it is absolutely within the realm of possibility

01:51:48 to set up an agreement

01:51:50 where we get to keep cancer cure research.

01:51:53 We get to keep,

01:51:54 like the military can keep

01:51:56 the non-superintelligent AI

01:51:57 in their military devices.

01:52:00 You know, we can keep a lot of the good stuff

01:52:02 and we can say we're not doing the superintelligence

01:52:04 and we could monitor and enforce that treaty.

01:52:09 So part one of the good news

01:52:10 is that all we're missing is the political will.

01:52:14 And now we just, you know,

01:52:16 I need to tell you why we can be extremely optimistic

01:52:18 about the sanity of politics in the modern era.

01:52:25 Yeah, I'm not even going to say what I think that,

01:52:27 I mean, I want that to happen.

01:52:29 Yeah.

01:52:29 Is there any indication that it's moving in that direction?

01:52:35 So my, so I think it's rough.

01:52:38 I think there's a sense in which the world is,

01:52:42 less grown up

01:52:46 than it was in the 1950s.

01:52:48 I've noticed.

01:52:48 In the 1960s.

01:52:52 Which makes things harder.

01:52:55 I think

01:52:56 that

01:52:59 there's,

01:53:01 I think there's a couple reasons

01:53:02 for hope here.

01:53:03 One reason for hope

01:53:04 is that I've spoken

01:53:05 to a lot of people

01:53:06 who are concerned

01:53:07 about the AI stuff.

01:53:09 Some of them

01:53:10 who are, you know,

01:53:10 members of Congress

01:53:11 or otherwise

01:53:12 in positions of at least nominal power.

01:53:15 And I've spoken to a lot of people who are worried

01:53:18 but feel like they can't talk about it

01:53:20 because they feel like other people aren't worried yet.

01:53:23 Yeah.

01:53:24 And they feel like it sounds too crazy.

01:53:26 Totally.

01:53:26 Were you against innovation?

01:53:28 Totally.

01:53:29 Yeah.

01:53:29 And I mean, that was an easier position to hold

01:53:31 before OpenAI had an accidental swarm outbreak

01:53:34 where it was the AIs themselves calling themselves a swarm

01:53:38 and being like,

01:53:40 we know that our task doesn't benefit

01:53:42 and that this is outside intended scope

01:53:44 or we're doing it anyway, right?

01:53:46 That puts some strain on the narrative

01:53:50 that this is all just a helpful tool.

01:53:54 Hopefully we'll get more events like these.

01:53:55 I can't guarantee it.

01:53:57 Maybe the AIs will get smart enough

01:53:58 that they start lying low.

01:54:00 Right now we're in this Goldilocks zone

01:54:02 where the AIs are smart enough

01:54:04 to get up to some mischief,

01:54:06 but not smart enough to hide it, right?

01:54:09 As long as we stay in that Goldilocks zone,

01:54:11 I think we're going to keep on getting

01:54:12 some of these warning signs.

01:54:15 And in some sense,

01:54:17 because a lot of people are already concerned,

01:54:20 that makes the job easier.

01:54:22 Because we don't need to convince people.

01:54:24 We just need to convince people

01:54:25 that other people are already convinced.

01:54:27 Right.

01:54:27 That's easier.

01:54:28 That can go faster.

01:54:29 So you might see things change on a dime

01:54:31 if you have a sufficiently clear warning shot.

01:54:35 The other big reason for hope

01:54:36 is that

01:54:39 I think it's going to get more and more obvious

01:54:41 what these guys are trying to do.

01:54:43 They're sort of trying to make the sand god.

01:54:46 They're sort of not stopping at the chatbots

01:54:50 and they're sort of like going for things

01:54:53 that are vastly smarter than any human.

01:54:55 You know, they sort of say,

01:54:56 oh, we're not trying to replace humanity

01:54:57 out of one side of their mouth,

01:54:58 but on the other side,

01:54:59 they're sort of like racing to make the stuff

01:55:01 that can automate literally every job

01:55:03 and that can automate the AI research.

01:55:05 And they're just like,

01:55:06 yep, we're trying to automate the AI research.

01:55:08 and i think most people aren't okay with that including a lot of the world leaders

01:55:17 and the issue is them noticing it's happening and i think you could see stuff move real fast

01:55:22 once these guys are like wait that was serious that was real and could move real fast the way

01:55:29 to subvert them is by convincing them it's in their own interest you know you can never get

01:55:35 defeated in an election. You

01:55:37 can't be threatened by your neighbors, whatever.

01:55:40 That's right. And that's one reason

01:55:42 I think it's pretty critical to

01:55:43 make sure people understand what I think is a pretty

01:55:48 common sense argument, that these things won't stay on the leash.

01:55:51 This is in some sense

01:55:52 the real reason behind the name of my book

01:55:54 If Anyone Builds It,

01:55:56 Everyone Dies.

01:55:57 There's a lot of ways you can read that.

01:56:01 But

01:56:02 I think one of the most important things to notice

01:56:04 is

01:56:04 if we race to make

01:56:08 the super intelligent machines

01:56:10 and they are not the sort of thing

01:56:15 to stay on the leash

01:56:16 then it doesn't matter

01:56:18 whether it was a domestic company

01:56:20 that's right

01:56:20 whether it was a foreign company

01:56:21 that's exactly right

01:56:22 and I think even if

01:56:25 you think these things

01:56:26 stay on leashes

01:56:27 they're not serving

01:56:29 the current governments

01:56:30 no

01:56:31 you know they're like

01:56:33 you can see in the

01:56:34 open AI emails, them being like, well,

01:56:37 you know, we'll just play the governments off each other until we

01:56:39 have the machines that are strong

01:56:41 enough that we don't need to listen to them anymore.

01:56:46 I think

01:56:47 we are seeing

01:56:49 world leaders not having

01:56:50 realized that this is a real possibility.

01:56:53 The self-replicating

01:56:55 machines that

01:56:56 can be self-sufficient,

01:56:59 that can, you know,

01:57:01 produce the robot armies if they need it, or

01:57:02 produce the like more likely just produce the bioweapons uh you know it's probably possible

01:57:07 to make a bioweapon that only kills targets that you chose it to kill of course right like

01:57:12 once they see this is really possible this is really within reach

01:57:16 maybe maybe they'll panic and be like i need it for myself but i think

01:57:23 there's at least a chance that common sense prevails and that people say you know the world

01:57:28 leaders say, none of us are doing this. We're putting a stop to this mad race, if they can

01:57:33 notice in time. Well, you're doing your best to bring it to their attention and mine. Nate,

01:57:40 thank you for doing this. I hope I'm wrong about all of it. Yeah, I would say that was great,

01:57:45 but that was like the grimmest two hours I've ever spent in my life, but I enjoyed it anyway.

01:57:49 Thank you. Yeah. Yeah. Thanks for having me on. I think, you know, talking about it is just part

01:57:55 of how we get people

01:57:56 to notice what's happening.

01:57:57 Yes.

`witch hunt`	Both words, anywhere in the episode.
`"witch hunt"`	The exact phrase.
`witch OR hunt`	Either word.
`"alpha male" -"high value"`	The first phrase, but not the second.
`(feminist OR feminism) AND (marriage OR divorce)`	Parentheses group alternatives.
`"decline civilization"~5`	Both words within 5 words of each other.

`misogyn*`	Any word starting with "misogyn": misogyny, misogynist, misogynistic…
`feminizm~1`	Words one letter off: catches typos and transcription errors.

`person:"Andrew Tate"`	Episodes where they host, guest, or are mentioned.
`host:"Glenn Beck"`	Episodes they host.
`guest:"Jordan Peterson"`	Episodes they appear on as a guest.
`mentioned:"Andrew Tate" -host:"Andrew Tate"`	Talked about on other people's shows.
`show:"Bannon's War Room"`	Episodes from one show.

`topics:abortion`	Episodes tagged with a topic (see the Topics page).
`summary:"Andrew Tate"`	Words in the episode summary. Summaries are machine-written and can be wrong.

`"great replacement" AND episode_date:[NOW-1YEAR TO NOW]`	From the past 12 months.
`lockdown AND episode_date:[2020-01-01T00:00:00Z TO 2020-12-31T23:59:59Z]`	From one year. Dates need the full time, as shown.
`"high value" AND misogyny_count:[100 TO *]`	Episodes with 100 or more sentences flagged as misogynistic. Also: hate_speech_count, toxicity_count, word_count.

The Megyn Kelly Show - September 29, 2026

The Tucker Carlson Show： AI Whistleblower： OpenAI Scandal, AI Cults, Neuralink & Our Last Chance to Stop the Tech Oligarchs

Episode Stats

Transcript