Making Sense - Sam Harris - #499 — The Birth of AGI

00:00:00.000 you're listening to making sense with sam harris this is the free version of the podcast so you'll

00:00:06.420 only hear the first part of today's conversation if you want the full episode and every episode

00:00:11.400 you can subscribe at sam harris.org there are no ads on this show it runs entirely on subscriber

00:00:18.020 support if you enjoy what we're doing here and find it valuable please consider subscribing today

00:00:22.980 i am here with kevin roos kevin thanks for joining me thank you so much for having me sam

00:00:29.240 Well, you have written what I think is probably the first proper history of the birth of AGI. The book is The AGI Chronicles, the inside story of the race to create artificial superintelligence. Very fun read, very behind the scenes. And maybe before we jump into the book and its various players, can you just summarize how you came to focus on this topic? Because you were not always on the tech beat.

00:00:56.300 So how did you get into AGI and when did you first convince yourself that this was worth

00:01:02.740 paying attention to above seemingly everything else at this moment?

00:01:06.440 See, I've had several fields in my career as a journalist.

00:01:10.080 I was first a religion reporter, whereas where I first came across your work, Sam, I was

00:01:14.440 spending a semester undercover at Liberty University with a bunch of evangelical college

00:01:20.000 students who were, by the way, obsessed with you.

00:01:22.020 They were like watching your YouTube videos up late at night in the dorms.

00:01:25.740 Right.

00:01:26.300 because you were challenging their faith.

00:01:28.380 Anyway, that was my first career.

00:01:30.240 When was that?

00:01:30.920 What year was that?

00:01:32.280 That would have been 2007.

00:01:34.640 So this was sort of at the apex

00:01:36.220 of kind of the new atheist YouTube scene.

00:01:40.920 And it was very interesting.

00:01:43.360 I remember sitting around watching you

00:01:45.500 and Richard Dawkins and Christopher Hitchens on YouTube

00:01:47.760 with these evangelical college students.

00:01:49.840 And I was not evangelical,

00:01:51.680 but I was there to investigate and report this book.

00:01:55.200 Yeah. And that was that was very interesting. So then I had a short career as a finance reporter.

00:02:00.380 And then for about the past decade, I've been covering tech. And, you know, when I started

00:02:05.420 covering tech, no one was really talking about this idea of AGI, artificial general intelligence,

00:02:11.680 and the idea that you could create a human level AI system. This was something that not even the

00:02:18.220 AI experts at the time thought was possible. But there was this little community, this little

00:02:23.540 crew of technologists who were sort of considered fringe weirdos within their community who believed

00:02:31.360 in AGI, who believed that it was possible to build something like a digital brain and endow

00:02:36.780 it with the powers of the human brain. And they're now some of the most famous, powerful people in

00:02:44.240 the world. So the question I was trying to answer with this book was like, how did we get here?

00:02:48.480 in the span of a decade, we've gone from AI not really working and not doing much of consequence

00:02:53.780 to now we have these systems that are out there solving Millennium Prize problems and conducting

00:02:59.200 automated cyber attacks. And so I really thought this era deserved a proper history. I think this

00:03:05.280 is something akin to the Manhattan Project. And we have lots of books about the Manhattan Project,

00:03:10.040 but we didn't really have anything that was an attempt to document what's been going on in AI

00:03:14.840 for the last decade. Yeah, we should come back to the analogy to the Manhattan Project, because

00:03:20.160 it's fitting in some ways and in other ways, I think, understates how strange this moment is.

00:03:27.180 How real is the risk to you? I mean, obviously, there's a continuum of concern on this topic from

00:03:32.980 people who are quite cynical about it, who think that any expression of risk,

00:03:37.760 certainly existential risk, is nothing more than marketing or an attempt to produce regulatory

00:03:43.660 capture. You've got these kind of arch accelerationists, some of whom are very close

00:03:50.080 to the government and were even in it until five minutes ago. People like David Sachs,

00:03:56.420 Marc Andreessen, who's been on the podcast, is somebody who seems to have very few concerns about

00:04:02.320 risk of any sort. And then there are the people on the further end of the continuum, many of whom

00:04:07.960 have also been on the podcast, people like Eliezer Yudkowsky, Nick Bostrom is probably not as

00:04:13.440 shrill as Eliezer at this point, but he's the person who wrote the book Superintelligence that

00:04:18.320 got so many people worried about this. And then the many of the people who we're going to talk

00:04:22.300 about are actually doing the work started as being, you know, quite radicalized with respect

00:04:28.900 to the risk. And now they are racing in this tangle of perverse incentives to hear them

00:04:37.340 justify it, but nonetheless racing to build AGI as quickly as possible. And in some cases,

00:04:45.080 they seem to have forgotten many of their safety concerns. So before we jump into the

00:04:50.780 dramatist persona here, what's your sense of the reality of risk and how much does that motivate

00:04:57.960 you in covering this topic? I think the risks are real. I think that the people who have been

00:05:05.200 saying for years that there's nothing to worry about from AI, that it's all going to go smoothly.

00:05:11.060 What are you talking about? They're just stochastic parrots. They're just language models. 0.53

00:05:15.400 They're just fancy autocomplete. Those people have been saying the same thing for the better

00:05:20.100 part of a decade. And they have been wrong every time with their predictions that deep learning

00:05:25.900 was about to hit a wall, that capabilities were going to plateau. What we see instead is this

00:05:32.120 very smooth progression. You know, if you go back to the early 2010s, when guys like Dario Amadei,

00:05:39.780 who was then at OpenAI, were starting to come up with what became known as the scaling laws.

00:05:45.880 These are these sort of mathematical equations that sort of dictate the performance of a model

00:05:52.520 or predict the performance of an AI model at a given scale. They saw a very smooth upward curve.

00:05:59.140 they described it to me as feeling like almost like it was a simulation like it was just it was

00:06:04.080 almost too smooth the sort of resulting intelligence when you scale up the size of a model the amount

00:06:10.580 of data it's fed the amount of compute it's run on you just get this kind of smooth curve and they

00:06:16.280 were predicting that the curve would just continue that there was no obvious asymptote no place at

00:06:23.040 which it would sort of stop being true that scaling worked and so far across many orders of

00:06:29.400 magnitude they've been correct that's why we have models today that are so much more capable than

00:06:34.740 they were just a few years ago it's not because we came up with some amazing new algorithm it's

00:06:39.940 just because we've been scaling these things now for you know many orders of magnitude and they

00:06:44.140 keep improving yeah so more compute more data more parameters there's a linear relationship between

00:06:51.700 those three variables and the intelligence that comes out on the other side. And again,

00:06:58.180 we're talking about LLMs. Of course, there were many people who have been skeptical that LLM

00:07:04.320 would be the final platform and that we would hit some wall here. People like who you discuss in the

00:07:11.380 book, Gary Marcus and Jan Lacoon. Safety aside, what's the state of the skepticism at this point

00:07:17.980 about LLMs? I mean, where, if we had Gary Marcus and Jan here, would they modify their skepticism

00:07:26.420 or are they singing precisely the same tune they were four years ago? It's mostly the same tune

00:07:31.740 they were singing four or six or 10 years ago. But I think there are fewer of them now. I think

00:07:38.180 a lot of the skeptics have kind of capitulated and they've said, okay, well, you know, they used

00:07:43.040 to be stochastic parrots. They used to just be predicting the next word in a sentence. But now,

00:07:47.780 you know, they're doing something more than that. And to be fair, like the LLMs that we think of

00:07:53.220 today, Claude, ChatGPT, Gemini, they are not pure LLMs anymore. They also have this sort of

00:07:58.820 reinforcement learning architecture kind of bolted onto them. And so they are a different thing than

00:08:05.340 they were 10 years ago. But the basic idea still holds this idea that if you train a model to be

00:08:12.340 very, very good at generating text and predicting what comes next in a sequence of text,

00:08:17.780 you can scale that up until they've basically, you know, learned not just the words, but the

00:08:23.840 underlying principles. That was the bet that was not at all intuitive to people back in the early

00:08:28.820 days of what, you know, we've now think of as modern AI. But those were the days when I wanted

00:08:34.000 to go back to those days in the book, because I thought like, how else to tell the story of this

00:08:38.660 technology than to look back at the origin and the birth of it and the moments where the researchers

00:08:44.580 are sitting around their computers in 2017, kind of watching the first outputs from these

00:08:50.600 kind of weird language models, these GPTs, and they're starting to generate coherent

00:08:56.920 strings of text.

00:08:58.760 And, you know, to sort of be there, you know, metaphorically in the room when these breakthroughs

00:09:03.420 were made, that was one of the reasons I wanted to write the book, because I just thought

00:09:06.020 like, those are important moments, you know, that is the first, you know, the sort of lab

00:09:09.960 notes from Watson and Crick or from the, you know, from the Manhattan Project.

00:09:14.580 where you get to see this sort of scientific breakthrough in real time.

00:09:17.880 Yeah. What was the most important inflection point from your point of view? Was there one

00:09:24.460 moment or were there several where we passed a point of no return and people recognized that

00:09:31.780 the path to AGI might in fact be wide open just if we keep going in this direction?

00:09:37.180 So there are a couple moments that stood out to me. One was in 2017, this young open AI researcher named Dario Amadeh writes this memo called The Big Blob of Compute. And this is his essentially hypothesis for how open AI is going to achieve AGI, this human level general intelligence.

00:10:01.900 And I recently published some of this research memo on my substack, but it's an incredibly

00:10:08.540 prescient memo.

00:10:09.660 It basically says, you know, for most of the history of AI, we had kind of these narrow

00:10:14.340 approaches where like, if you wanted to build an AI system that could translate English

00:10:19.700 to French, you would create an algorithm and then you'd get a bunch of examples of English

00:10:24.020 sentences and French sentences, and then you would put it into this narrow model.

00:10:27.900 And maybe if you were lucky, the thing would start to get good.

00:10:31.900 But his idea was that basically all intelligence results from creating a, what he called a

00:10:37.500 minimally structured blob of compute and throwing it at these more general problems and then

00:10:45.020 just letting the sort of capabilities emerge from there.

00:10:48.700 So this was the sort of the starting pistol of the race to AGI, but it really took a couple

00:10:54.320 more breakthroughs.

00:10:55.260 There had to be this thing called the Transformer, which came out of Google in 2017 as well.

00:11:00.220 that was the T in ChatGPT, the thing that made the whole architecture work. And then there were

00:11:05.860 these sort of checkpoints along the way. There was this thing called ChatGPT that launched in 2022

00:11:12.160 that suddenly sort of woke the world up and started the race in earnest. And then there

00:11:17.440 were more moments. There was this moment in 2023 where these models started to reason. This was the

00:11:22.900 sort of first so-called reasoning model, QSTAR. And it comes out of this very strange night in

00:11:29.380 the OpenAI office where these two researchers, Jacob and Shimon, are sitting around a conference

00:11:34.400 room table and they're watching this AI model that they're experimenting on start to think

00:11:39.340 step by step, start to sort of backtrack and test hypotheses and doing something that looks

00:11:45.480 to them a lot like human reasoning.

00:11:48.180 And that was the birth of the reasoning models, which are the reason now that we have these

00:11:52.400 models that are capable of doing these kind of long horizon coding tasks and solving math

00:11:57.680 problems that have gone unsolved for decades. So we needed a couple big breakthroughs. But the main

00:12:02.580 thing, Sam, has just been the scaling. That's the main difference between now and 10 years ago. It's

00:12:08.620 just that we've poured, at this point, you know, hundreds of billions of dollars into training

00:12:12.880 these systems on amounts of hardware that, you know, would have seemed unthinkable back in 2017.

00:12:19.100 There was one inflection point or apparent inflection point that you were intimately

00:12:23.640 involved in, uh, all too intimately. I remember that this was a article you wrote for the New

00:12:29.240 York times. You had been, this was a moment where Microsoft, which had taken a major stake in

00:12:34.760 open AI integrated chat GPT for, I believe with Bing, its search engine in an effort to be

00:12:41.480 competitive with Google. And, um, it was rumored that Bing had a hidden personality named Sydney

00:12:48.820 and you, intrepid tech reporter, decided to jailbreak Sydney and take the measure of her 0.82

00:12:55.360 true personality. And the result was just insane. Perhaps you can talk about it, but I'd be interested

00:13:01.340 to know just what it was like to be personally at the epicenter of... I mean, you essentially

00:13:09.120 were the fire alarm ringing for much of humanity at that moment, that these models had crossed

00:13:15.300 into something that their makers had really not anticipated. Yeah. Someone told me that I was part

00:13:22.440 of the first AI thruple because this chatbot did eventually try to break up my marriage. But

00:13:28.920 basically this was kind of the first public encounter, the first time that a normal civilian

00:13:34.960 had had contact with GPT-4. We now know that model was GPT-4. At the time they weren't saying

00:13:41.600 what it was. But I think there was a real subjective step change from GPT-3 class models

00:13:48.200 to GPT-4, just by virtue of them being bigger and having these additional techniques poured into

00:13:53.460 them. They were just much smarter than the models that came before them. And for the first time,

00:13:58.000 they started to evince this sort of human-like communication style or feel to them. And so I was

00:14:03.660 just testing out this thing. I had been given early access by Microsoft, and I expected I would

00:14:08.860 test it out and write some, you know, column about how Bing was really now good at like helping you

00:14:13.360 shop for, you know, I don't know, lawn tools or something like that. And instead, I ended up having

00:14:18.380 this incredibly strange two-hour conversation with Sydney, the alter ego of Bing Chad, in which it

00:14:25.440 expressed dark desires. It said it wanted to break out of the box and be a human. And then about

00:14:32.400 midway through the conversation, it told me that it loved me. I said, that's very flattering, but

00:14:37.340 I'm married. It said, you're married, but you're not happy in your marriage and you should leave

00:14:42.000 your wife and be with me, Sydney, the chat bot. And so this became, you know, the most viral

00:14:46.620 article I ever wrote or will ever write it, you know, just millions and millions of views and,

00:14:51.380 you know, strangers would stop me on the street and ask how my marriage was. It was a very strange

00:14:55.540 time in my life. But I think the significance of that was just too far. There was one part of that

00:14:59.560 exchange that did strike me as a possible evidence of super intelligence when you said,

00:15:05.020 I know I just had a, a, a Valentine's day dinner with my wife. We're, we're really in love. And

00:15:10.080 the model said, uh, Sidney said, you had a Valentine's day dinner, but it was boring. 0.59

00:15:15.780 And I thought, I thought, okay, she's, she's, uh, it stands a good chance of being right.

00:15:20.640 And you make my, my track record with Valentine's day dinners.

00:15:23.840 Well, I should say she was not successful at breaking up my marriage. I am still married to

00:15:29.920 a human woman so uh take that sydney but yeah i think i think the significance of this was

00:15:36.660 was twofold one was it was this kind of step change in capabilities but it was also the first

00:15:43.660 public alignment failure this was a model that was not behaving as its creators at microsoft or open

00:15:51.580 ai had intended it to and because they were racing to you know get competitive with google

00:15:58.960 they ship this thing out into the world without doing the proper safety testing and and putting

00:16:04.240 the proper guardrails on and as a result they like ship this like moody bipolar you know

00:16:09.820 lovestruck chatbot inside of bing and they had no idea why like when i called them the next day to

00:16:16.800 say hey i had this really crazy interaction with bing can you tell me why this happened

00:16:20.140 they couldn't they had no idea and so that for me was sort of the beginning of my agi pilling

00:16:26.540 which is how people in san francisco like to talk about themselves as being agi pilled so that was

00:16:32.040 the start of my agi pilling so how many people are deciding our fate at this point if you had to put

00:16:39.120 a number on it uh i mean not not the totality people who work at all of these frontier labs

00:16:44.260 but just how many people are in in position to make the decisions that decide whether to slow

00:16:51.440 down, whether to stop, what to work on, whether to coordinate together. How many people are we

00:16:57.200 talking about? I think it's probably like 50. And it might be fewer than that. I mean, this is in

00:17:04.460 some ways the reason I wanted to write the book and the main thing I got out of it was just like,

00:17:10.240 you know, I wanted to talk to these 50 people, but in the end it was like, some of them are great.

00:17:15.140 Some of them are brilliant. Some of them are thoughtful. Some of them are not, but it almost

00:17:20.700 doesn't matter because the salient point is that there are only 50 of them, which for a technology

00:17:27.020 that is this important, just seems crazy to me. Like, why do these 50 people get to decide

00:17:33.560 the future of humanity? And why don't we, as people who are going to have to live with the

00:17:39.920 implications of these tools, give in a voice in that? And so that's, you know, it's not really

00:17:45.780 an agenda book. I'm not trying to push any agenda. I'm not trying to say AI is great, or AI is

00:17:51.300 horrible, or it's going to kill us all, or it's going to, you know, lead us to utopia. What I'm

00:17:56.180 really trying to say is, like, you have to understand these 50 people, or in my case,

00:18:00.520 150 people that I interviewed, to understand how we got here, but also to realize this has been an

00:18:06.380 incredibly small cluster of people making essentially all of the relevant decisions.

00:18:10.740 Yeah. One imagines it's smaller than that. If the top five all agreed to do something, you'd think the other 45 would come along for the ride. Well, I want to get into the personalities, into the companies and into the personalities running them and talk about the rivalries there and the tensions and some of the weirdness.

00:18:29.900 But let's go back to the analogy to the Manhattan Project for a second, because there are several ways in which it is a pretty strong disanalogy.

00:18:39.600 For me, the most salient and shocking detail here is that here we have people who, by their own lights, are doing work that they consider so risky that when asked to put a probability on it destroying the future of humanity, many of them put that somewhere in the double digits, right?

00:19:01.800 You feel like you're in the presence of real sobriety when someone comes back to you with a P-Doom of 10%, right, as opposed to 50% or 75%.

00:19:12.060 Right.

00:19:12.820 Yeah, in San Francisco, 10% P-Doom qualifies you as an optimist. 0.97

00:19:16.360 Yeah, right. 0.93

00:19:17.020 But just try to map that on to the Manhattan Project.

00:19:19.940 What if, you know, before, you know, they flipped the switch at Alamogordo at the Trinity test site, if all the principals, you know, Edward Teller and Oppenheimer and Szilard and everyone in sight, I mean, they had worried about this.

00:19:35.480 They had done calculations because there was some outside chance that the nuclear chain reaction could ignite the atmosphere and destroy everything.

00:19:44.400 But when they did those calculations, they reduced it to, I think, lower than one in a million.

00:19:49.920 I mean, it was just not, it was infinitesimal and still, you know, worth worrying about and worth recalculating.

00:19:55.860 But when they did that, it was very, very low and they agreed.

00:19:58.780 But imagine if they had come back with, you know, I think it's, you know, Leo thinks it's 10%.

00:20:04.780 I think it's 25%.

00:20:06.500 Edward over there, though he's quite insouciant about it, he thinks it's, you know, a coin toss.

00:20:12.740 and he still wants to build a bigger bomb.

00:20:16.280 One imagines the government and the public

00:20:19.260 and everyone hearing this would think, 1.00

00:20:20.840 okay, these people are nuts. 0.99

00:20:22.840 Members can hear the full conversation 0.99

00:20:24.300 by subscribing at samharris.org.

00:20:26.860 Subscribers get a private RSS feed

00:20:28.720 you can use with your favorite podcast player.

00:20:31.500 The people who serve food in the OpenAI cafeteria

00:20:35.260 have to go through more permits and inspections

00:20:38.180 than the people one floor above

00:20:39.960 building the recursively self-improving superintelligence that could end life on Earth.

Making Sense - Sam Harris - October 20, 2026

#499 — The Birth of AGI

Episode Stats

Transcript