The Joe Rogan Experience - Joe Rogan Experience #2156

00:00:01.000 Joe Rogan Podcast, check it out!

00:00:04.000 The Joe Rogan Experience.

00:00:06.000 Train by day, Joe Rogan Podcast by night, all day.

00:00:14.000 Oh, you know, not too much.

00:00:16.000 Just another typical week in AI. Just the beginning of the end of time.

00:00:21.000 That's all happening right now.

00:00:22.000 Just for the sake of the listeners, please just give us your names and tell us what you do.

00:00:29.000 So, I'm Jeremy Harris.

00:00:30.000 I'm the CEO and co-founder of this company, Gladstone AI, that we co-founded.

00:00:34.000 So, we're essentially a national security and AI company.

00:00:37.000 We can get into the backstory a little bit later, but that's the high level.

00:00:40.000 Yeah.

00:00:41.000 Yeah.

00:00:41.000 And I'm Ed Harris.

00:00:42.000 I'm actually, I'm his co-founder and brother and the CTO of the company.

00:00:48.000 Keep this, like, pull this up like a fist from your face.

00:00:51.000 There you go.

00:00:52.000 Perfect.

00:00:53.000 So, how long have you guys been involved in the whole AI space?

00:00:59.000 For a while, in different ways.

00:01:01.000 We started off as physicists.

00:01:03.000 That was our background.

00:01:06.000 Around 2017, we started to go into AI startups.

00:01:10.000 We founded a startup, took it through Y Combinator, this Silicon Valley accelerator program.

00:01:15.000 At the time, actually, Sam Altman, who's now the CEO of OpenAI, was the president of Y Combinator.

00:01:20.000 He opened up our batch at YC with this big speech, and we got some conversations in with him over the course of the batch.

00:01:28.000 In 2020, so this thing happened that we could talk about.

00:01:32.000 Essentially, this was like the moment that there's like a before and after in the world of AI, before and after 2020. And it launched this revolution that brought us to ChatGPT.

00:01:43.000 Essentially, there was an insight that OpenAI had and doubled down on that you can draw a straight line to ChatGPT, GPT-4, Google Gemini, everything that makes AI everything it is today started then.

00:01:54.000 And when it happened, we kind of went...

00:01:57.000 Well, Ed gave me a call, this, like, panicked phone call.

00:01:59.000 He's like, dude, I don't think we can keep working, like, business as usual in a regular company anymore.

00:02:06.000 Yeah.

00:02:07.000 So there was this AI model called GPT-3.

00:02:10.000 So, like, everyone has, you know, maybe played with GPT-4.

00:02:13.000 It's like ChatGPT.

00:02:15.000 GPT-3 was the generation before that.

00:02:19.000 And it was the first time that you had an AI model that could...

00:02:24.000 We're good to go.

00:02:51.000 I think?

00:03:06.000 And the significance of that is you increase the amount of computing cycles you put against something.

00:03:12.000 You increase the amount of data.

00:03:14.000 All of that is an engineering problem and you can solve it with money.

00:03:19.000 So you can scale up the system, use it to make money, and put that money right back into scaling up the system some more.

00:03:26.000 Money in, IQ points come out.

00:03:29.000 Jeez.

00:03:29.000 That was kind of the 2020 moment.

00:03:31.000 And that's what we said in 2020, exactly.

00:03:34.000 I spent about two hours trying to argue him out of it.

00:03:36.000 I was like, no, no, no, we can keep working at our company because we're having fun.

00:03:39.000 We like founding companies.

00:03:41.000 And yeah, he just wrestled me to the ground and we're like, shit, we got to do something about this.

00:03:45.000 We reached out to a family friend who, he was non-technical, but he had some connections in government in DOD and we're like, dude, the way this is set up right now, You can really start drawing straight lines and extrapolating and saying you know what the government is going to give a shit about this in Not very long,

00:04:06.000 two years, four years, we're not sure, but the knowledge about what's going on here is so siloed in the Frontier Labs.

00:04:13.000 Like, our friends are, you know, all over the Frontier Labs, the OpenAI's, Google DeepMind's, all that stuff.

00:04:17.000 The shit they were saying to us that was like mundane reality, like water cooler conversation, when you then went to talk to people in policy and even like pretty senior people in government, Not tracking the story remotely.

00:04:30.000 In fact, you're hearing almost the diametric opposite.

00:04:33.000 This is sort of like over learning the lessons of the AI winters that came before when it's pretty clear like we're on a very at least interesting trajectory, let's say, that should change the way we're thinking about the technology.

00:04:44.000 What was your fear?

00:04:46.000 Like what was it that hit you that made you go, we have to stop doing this?

00:04:50.000 So it's basically, you know, anyone can draw a straight line, right, on a graph.

00:04:57.000 The key is looking ahead and actually at that point, three years out, four years out, and asking like you're asking, what does this mean for the world?

00:05:07.000 What does it mean?

00:05:07.000 What does the world have to look like if we're at this point?

00:05:12.000 And We're already seeing the first kind of wave of risk sets just begin to materialize, and that's kind of the weaponization risk sets.

00:05:22.000 So you think about stuff like large-scale psychological manipulation of social media.

00:05:28.000 Actually, really easy to do now.

00:05:30.000 You train a model on just a whole bunch of tweets.

00:05:33.000 You can actually direct it to push a narrative like, you know, maybe China should own Taiwan or, you know, whatever, something like that.

00:05:40.000 And you actually, you can train it to adjust the discourse and have increasing levels of effectiveness to that.

00:05:48.000 Just as you increase the general capability surface of these systems, we don't know how to predict what exactly comes out of them at each level of scale.

00:05:59.000 But it's just general increasing power.

00:06:01.000 And then the kind of next beat of risk after that So we're scaling these systems.

00:06:10.000 We're on track to scale systems that are at human level, like generally as smart, however you define that as a person or greater.

00:06:19.000 And OpenAI and the other labs are saying, yeah, it might be two years away, three years away, four years away, like insanely close.

00:06:27.000 At the same time, and we can go into the details of this, but we actually don't understand how to reliably control these systems.

00:06:42.000 What it is we want.

00:06:43.000 We can kind of like poke them and prod them and get them to kind of adjust.

00:06:46.000 But you've seen, and we can go over these examples, we've seen example after example of, you know, Bing Sydney yelling at users, Google showing 17th century British scientists that are racially diverse, all that kind of stuff.

00:06:59.000 We don't really understand how to like aim it or align it or steer it.

00:07:03.000 And so then you can ask yourself, well, we're on track to get here.

00:07:08.000 We are not on track to control these systems effectively.

00:07:11.000 How bad is that?

00:07:30.000 Now, when a system does something like what Gemini did, like it says, show us Nazi soldiers and it shows you Asian women, what's the mechanism?

00:07:42.000 How does that happen?

00:07:43.000 So it's maybe worth taking a step back and looking at how these systems actually work.

00:07:48.000 Because that's going to give us a bit of a frame, too, for figuring out when we see weird shit happen, How weird is that shit?

00:07:55.000 Is that shit just explainable by just the basic mechanics of what you would expect to happen based on the way we're training these things?

00:08:01.000 Or is something new and fundamentally different happening?

00:08:03.000 So we're talking about this idea of scaling these AI systems, right?

00:08:07.000 What does that actually mean?

00:08:08.000 Well, you imagine the AI model, which is kind of like you think of it as like the artificial brain here that actually does the thinking.

00:08:15.000 That model contains, it's kind of like a human brain, it's got these things called neurons.

00:08:19.000 We, in the human brain, call them biological neurons in the context of AI, it's artificial neurons, but it doesn't really matter.

00:08:24.000 They're the cells that do the thinking for the machine.

00:08:27.000 And the realization of AI scaling is that you can basically take this model, increase the number of artificial neurons it contains, And at the same time, increase the amount of computing power that you're putting into kind of like wiring the connections between those neurons.

00:08:41.000 That's the training process.

00:08:42.000 Can I pause you right there?

00:08:43.000 Yeah.

00:08:43.000 How does the neuron think?

00:08:46.000 Yeah.

00:08:47.000 Okay.

00:08:47.000 So let's get a little bit more concrete then.

00:08:49.000 So in your brain, right, we have these neurons.

00:08:51.000 They're all connected to each other with different connections.

00:08:54.000 And when you go out into the world and you learn new skill, what really happens is you try out that skill, you succeed or fail, and based on your succeeding or failing, the connections between neurons that are associated with doing that task well get stronger.

00:09:09.000 The connections that are associated with doing it badly get weaker.

00:09:13.000 And over time, through this like glorified process really of trial and error, eventually you're going to hone in and really in a very real sense, everything you know about the world gets implicitly encoded in the strengths of the connections between all those neurons.

00:09:28.000 If I can x-ray your brain and get all the connection strengths of all the neurons, I have everything Joe Rogan has learned about the world.

00:09:35.000 That's like basically the...

00:09:37.000 A good sketch, let's say, of what's going on here.

00:09:40.000 So now we apply that to AI, right?

00:09:42.000 That's the next step.

00:09:44.000 And here, really, it's the same story.

00:09:45.000 We have these massive systems, artificial neurons connected to each other.

00:09:50.000 The strength of those connections is secretly what encodes all the knowledge.

00:09:53.000 So if I can steal all of those connections, those weights, as they're sometimes called, I've stolen the model.

00:09:59.000 I've stolen the artificial brain.

00:10:00.000 I can use it to do whatever the model could do initially.

00:10:03.000 That is kind of the artifact of central interest here.

00:10:06.000 And so, if you can build the system, right, now you've got so many moving parts.

00:10:11.000 Like if you look at GPT-4, it has people think around a trillion of these connections.

00:10:16.000 And that's a trillion little pieces that all have to be jiggered together to work together coherently.

00:10:21.000 And you need computers to go through and like tweak those numbers.

00:10:24.000 So massive amounts of computing power.

00:10:26.000 The bigger you make that model, the more computing power you're going to need to kind of tune it in.

00:10:31.000 And now you have this relationship between the size of your model, the amount of computing power you're going to use to train it, and if you can increase those things at the same time, what Ed was saying is your IQ points basically drop out.

00:10:42.000 Very roughly speaking, that was what people realized in 2020. And the effect that had, right, was now all of a sudden the entire AI industry is looking at this equation.

00:10:51.000 Everybody knows the secret sauce.

00:10:52.000 I make it bigger.

00:10:53.000 I make more IQ points.

00:10:55.000 I can get more money.

00:10:56.000 So Google's looking at this.

00:10:57.000 Microsoft, OpenAI, Amazon.

00:10:59.000 Everybody's looking at the same equation.

00:11:01.000 You have the makings for a crazy race.

00:11:04.000 Like, right now today, Microsoft is engaged in the single biggest infrastructure in human history.

00:11:11.000 The biggest infrastructure build-out.

00:11:13.000 $50 billion a year, right?

00:11:15.000 So on the scale of the Apollo moon landings, just in building out data centers to house the compute infrastructure, because they are betting that these systems are going to get them to something like human-level AI pretty damn soon.

00:11:29.000 So, I was reading some story about, I think it was Google, that's saying that they're going to have multiple nuclear reactors to power their database?

00:11:39.000 That's what you've got to do now, because what's going on is North America is kind of running out of on-grid...

00:11:47.000 Baseload power to actually supply these data centers.

00:11:51.000 You're getting data center building moratoriums in areas like Virginia, which has traditionally been the data center cluster for Amazon, for example, and for a lot of these other companies.

00:12:04.000 And so when you build a data center, you need a bunch of resources, you know, sited close to that data center.

00:12:10.000 You need water for cooling and a source of electricity.

00:12:13.000 And it turns out that, you know, wind and solar don't really quite cut it for these big data centers that train big models.

00:12:20.000 Because the data center, the training consumes power like this all the time.

00:12:26.000 But the sun isn't always shining.

00:12:27.000 The wind isn't always blowing.

00:12:28.000 And so...

00:12:29.000 You've got to build nuclear reactors, which give you high-capacity factor baseload.

00:12:35.000 And Amazon literally bought, yeah, a data center with a nuclear plant right next to it.

00:12:40.000 Because, like, that's what you've got to do.

00:12:42.000 Jesus.

00:12:44.000 How long does it take to build a nuclear reactor?

00:12:47.000 Because, like, this is the race, right?

00:12:49.000 The race is...

00:12:50.000 You're talking about 2020, people realizing this.

00:12:54.000 Then you have to have the power to supply it.

00:12:57.000 But how long, how many years does it take to get an active nuclear reactor up and running?

00:13:02.000 It's an answer that depends.

00:13:06.000 The Chinese are faster than us at building nuclear reactors, for example.

00:13:09.000 And that's part of the geopolitics of this too, right?

00:13:11.000 Yeah.

00:13:14.000 What is bottlenecking each country, right?

00:13:16.000 So the US is bottlenecked increasingly by power, baseload power.

00:13:20.000 China, because we've got export control measures in place, in part as a response to the scaling phenomenon.

00:13:26.000 And as a result of the investigation we did.

00:13:28.000 That's right, yeah.

00:13:30.000 In part.

00:13:31.000 In part, yeah.

00:13:32.000 But China is bottlenecked by their access to the actual processors.

00:13:36.000 They've got all the power they can eat because they've got much more infrastructure investment, but the chip side is weaker.

00:13:42.000 So there's this sort of balancing act between the two sides, and it's not clear yet which one positions you strategically for dominance in the long term.

00:13:49.000 But we are also building better So small modular reactors, essentially small nuclear power plants that can be mass produced.

00:13:58.000 Those are starting to come online relatively early, but the technology and designs are pretty mature.

00:14:02.000 So that's probably the next beat for our power grid for data centers, I would imagine.

00:14:08.000 Microsoft is doing this.

00:14:09.000 So in 2020, you have this revelation.

00:14:13.000 You recognize where this is going.

00:14:15.000 You see how it charts, and you say, this is going to be a real problem.

00:14:19.000 Does anybody listen to you?

00:14:22.000 This is where the problem comes, right?

00:14:24.000 Yeah, like we said, right?

00:14:25.000 You can draw a straight line.

00:14:27.000 You can have people nodding along, but there's a couple of hiccups along the way.

00:14:33.000 One, is that straight line really going to happen?

00:14:36.000 All you're doing is drawing lines on charts, right?

00:14:38.000 I don't really believe that that's going to happen, and that's one thing.

00:14:41.000 The next thing is just imagining, is this what's going to come to pass as a result of that?

00:14:47.000 And then the third thing is, well...

00:14:50.000 Yeah, that sounds important, but not my problem.

00:14:53.000 That sounds like an important problem for somebody else.

00:14:55.000 And so we did do a bit of a traveling...

00:14:59.000 Yeah, it was like the world's saddest traveling roadshow.

00:15:01.000 It was literally as dumb as this sounds.

00:15:04.000 So we go and...

00:15:05.000 Oh my God.

00:15:06.000 I mean, it's almost embarrassing to think back on.

00:15:07.000 So 2020 happens, yes, within months.

00:15:10.000 First of all, we're like, we've got to figure out how to hand off our company.

00:15:13.000 So we handed it off to two of our earliest employees.

00:15:15.000 They did an amazing job.

00:15:16.000 The company exited.

00:15:17.000 That's great.

00:15:18.000 But that was only because they're so good at what they do.

00:15:21.000 We then went, what the hell?

00:15:23.000 How can you steer this situation?

00:15:26.000 We just thought we got to wake up the U.S. government.

00:15:28.000 As stupid and naive as that sounds, that was the big picture goal.

00:15:31.000 So we start to line up as many briefings as we possibly can across the U.S. interagency, all the departments, all the agencies that we can find, climbing our way up.

00:15:39.000 We got an awful lot, like Ed said, of like, that sounds like a wicked important problem for somebody else to solve.

00:15:45.000 Yeah, like defense, Homeland Security, and then the State Department.

00:15:48.000 Yeah, so we end up exactly in this meeting with, like, there's about a dozen folks from the State Department.

00:15:53.000 And one of them, and I hope at some point, you know, history recognizes what she did and her team did, because it was the first time that somebody actually stood up and said, first of all, yes, sounds like a serious issue.

00:16:06.000 I see the argument makes sense.

00:16:08.000 Two, I own this.

00:16:09.000 And three, I'm going to put my own career capital behind this.

00:16:13.000 That's the...

00:16:14.000 And that was at the end of 2021. So imagine that.

00:16:16.000 That's a year before ChatGPT.

00:16:18.000 Nobody was tracking this issue.

00:16:20.000 You had to have the imagination to draw through that line, understand what it meant, and then believe, yeah, I'm going to risk some career capital on this.

00:16:31.000 Yeah.

00:16:51.000 You know, not nothing burger investigation, but, you know, sure, go ahead.

00:16:55.000 And it just became this insane story.

00:16:57.000 We had, like, the UK AI Safety Summit.

00:16:59.000 We had the White House executive order.

00:17:01.000 All this stuff which became entangled with the work we were doing, which we simply could not have, especially some of the reports we were collecting from the labs, the whistleblower reports, that could not have been made public if it wasn't for the foresight of this team really pushing for, as well, the American population to hear about it.

00:17:17.000 I could see how if you were one of the people that's on this expansion-minded mindset, all you're thinking about is getting this up and running.

00:17:28.000 You guys are a pain in the ass, right?

00:17:31.000 So you guys, you're obviously, you're doing something really ridiculous.

00:17:35.000 You're stopping your company.

00:17:37.000 You can make more money staying there and continuing the process.

00:17:40.000 But you recognize that there's like an existential threat involved in making this stuff go online.

00:17:47.000 Like when this stuff is live, you can't undo it.

00:17:50.000 Oh yeah, I mean like no matter how much money you're making, The dumbest thing to do is to stand by as something that completely transcends money, is being developed, and it's just going to screw you over if things go badly.

00:18:00.000 My point is, are there people that push back against this, and what is their argument?

00:18:07.000 Yeah, so actually, and I'll let you follow up on that, but the first story of the pushback, I think it's been in the news a little bit lately now, getting more and more public, but When we started this, and no one was talking about it, the one group that was actually pushing stuff in this space was a big funder in the area of effective altruism.

00:18:30.000 You may have heard of them.

00:18:31.000 This is kind of a Silicon Valley group of people who have a certain mindset about how you pick tough problems to work on, valuable problems to work on.

00:18:39.000 They've had all kinds of issues.

00:18:40.000 Sam Bankman-Fried was one of them and all that, quite famously.

00:18:44.000 So we're not effective altruists, but because these are the folks who are working in the space, we decided, well, we'll talk to them.

00:18:49.000 And the first thing they told us was, don't talk to the government about this.

00:18:54.000 Their position was, if you bring this to the attention of the government, they will go, oh shit, powerful AI systems?

00:19:02.000 And they're not going to hear about the dangers, so they're going to somehow go out and build the powerful systems without caring about the risk side.

00:19:09.000 Which...

00:19:10.000 When you're in that startup mindset, you want to fail cheap.

00:19:14.000 You don't want to just make assumptions about the world and be like, okay, let's not touch it.

00:19:17.000 So our instinct was, okay, let's just test this a little bit and talk to a couple people, see how they respond, tweak the message, keep climbing that ladder.

00:19:26.000 That's the kind of builder mindset that we came from in Silicon Valley.

00:19:30.000 And we found that people are way more thoughtful about this than you would imagine.

00:19:34.000 In DOD especially.

00:19:35.000 DOD actually has a very safety-oriented culture with their tech.

00:19:39.000 The thing is, because their stuff kills people, right?

00:19:44.000 And they know their stuff kills people.

00:19:46.000 And so they have an entire safety-oriented development practice To make sure that their stuff doesn't like go off the rails.

00:19:53.000 And so you can actually bring up these concerns with them and it lands in kind of a ready culture.

00:19:59.000 But one of the issues with the individuals we spoke to who were saying don't talk to government is that they had just not actually interacted with I think?

00:20:33.000 Reality is, you could have made it your life's mission to try to get the Department of Defense to build an AGI and, like, you would not have succeeded because nobody was paying attention.

00:20:45.000 Wow.

00:20:47.000 Because they just didn't know.

00:20:48.000 Yeah, there's a chasm, right?

00:20:50.000 There's a gap to cross.

00:20:52.000 Like, there's information.

00:20:52.000 It's a cultural, yeah.

00:21:09.000 Constantly, you know, iterate on, like, clarity, making it very kind of clear and explaining it and all that stuff.

00:21:16.000 Years at times.

00:21:16.000 And that was the piece to your question about, like, the pushback from, in a way, from inside the house.

00:21:21.000 I mean, that was the people who cared about the risk.

00:21:24.000 Yeah.

00:21:25.000 Man, I mean, like, when we actually went into the labs.

00:21:29.000 So some labs, not all labs are created equal.

00:21:31.000 We should make that point.

00:21:33.000 You know, when you talk to whistleblowers, what we found was So there's one lab that's really great, so anthropic.

00:21:40.000 When you talk to people there, you don't have the sense that you're talking to a whistleblower who's nervous about telling you whatever.

00:21:46.000 Roughly speaking, what the executives say to the public is aligned with what their researchers say.

00:21:51.000 It's all very, very open.

00:21:53.000 More closely, I think, than any of the others.

00:21:55.000 Sorry, yeah, more closely than any of the others.

00:21:58.000 There are always variations here and there.

00:21:59.000 But some of the other labs, like, very different story.

00:22:02.000 And you had the sense, like, we were in a room with one of the frontier labs.

00:22:06.000 We're talking to their leadership as part of the investigation.

00:22:09.000 And there was somebody from...

00:22:11.000 Anyway, it won't be too specific, but there was somebody in the room who then took us aside after.

00:22:15.000 And he hands me his phone.

00:22:17.000 He's like, hey, can you please, like, put your phone number in...

00:22:20.000 Sorry, yeah, can you please put...

00:22:22.000 Or no, yeah, sorry, he put his number in my phone.

00:22:25.000 And then he kind of like whispered to me, he's like, hey, so whatever recommendations you guys are going to make, I would urge you to be more ambitious.

00:22:35.000 And I was like, what does that mean?

00:22:37.000 He's like, can we just talk later?

00:22:39.000 So as happened in many, many cases, we had a lot of cases where we set up bar meetups after the fact, where we would talk to these folks and get them in an informal setting.

00:22:51.000 He shared some pretty sobering stuff, and in particular, the fact that he did not have confidence in his lab's leadership to live up to their publicly stated word on what they would do when they were approaching AGI, and even now, to secure and make these systems safe.

00:23:06.000 So many such cases is like kind of one specific example, but it's not that you ever had like lab leadership come in or doors getting kicked down and people are waking us up in the middle of the night.

00:23:17.000 It was that you had this looming cloud over everybody that you really felt some of the people with the most access and information who understood the problem the most deeply were the most hesitant to bring things forward because they sort of understood that their labs not going to be happy with this.

00:23:33.000 And so it's very hard to also get an extremely broad view of this from inside the labs because, you know, you open it up, you start to talk to...

00:23:42.000 We spoke to like a couple of dozen people about various issues in total.

00:23:47.000 You go much further than that and, you know, word starts to get around.

00:23:51.000 And so we had to kind of strike that balance as we spoke to folks from each of these labs.

00:23:56.000 Now, when you say approaching AGI, how does one know when a system has achieved AGI and does the system have an obligation to alert you?

00:24:09.000 Well, by, you know the Turing test, right?

00:24:12.000 Yes.

00:24:13.000 Yeah.

00:24:13.000 So you have a conversation with a machine and it can fool you into thinking that it's a human.

00:24:19.000 We're good to go.

00:24:38.000 The definition of AGI itself is kind of interesting, right?

00:24:41.000 Because we're not necessarily fans of the term because usually when people talk about AGI, they're talking about a specific circumstance in which there are capabilities they care about.

00:24:52.000 So some people use AGI to refer to the wholesale automation of all labor, right?

00:24:56.000 That's one.

00:24:57.000 Some people say, well, when you build AGI, it's like it's automatically going to be hard to control and there's a risk to civilization.

00:25:04.000 So that's a different threshold.

00:25:06.000 And so all these different ways of defining it, ultimately, it can be more useful to think sometimes about advanced AI and the different thresholds of capability you cross and the implications of those capabilities.

00:25:17.000 But it is probably going to be more like a fuzzy spectrum, which in a way makes it harder, right?

00:25:21.000 Because it would be great to have like...

00:25:23.000 Like a tripwire where you're like, oh, this is bad.

00:25:26.000 Okay, we've got to do something.

00:25:29.000 But because there's no threshold that we can really put our fingers on, we're like a frog in boiling water in some sense, where it's like, oh, it just gets a little better, a little better.

00:25:39.000 Oh, we're still fine.

00:25:41.000 And not just we're still fine, but as the system improves below that threshold...

00:25:48.000 Life gets better and better.

00:25:50.000 These are incredibly valuable, beneficial systems.

00:25:53.000 We do roll stuff out like this, again, at DoD and various customers, and it's massively valuable.

00:26:01.000 It allows you to accelerate all kinds of, you know, back office, like paperwork, BS. It allows you to do all sorts of wonderful things.

00:26:10.000 And our expectation is that's going to keep happening until it suddenly doesn't.

00:26:16.000 Yeah, one of the things that there was a guy we were talking to from one of the labs, and he was saying, look, the temptation to put a heavier foot on the pedal is going to be greatest, just as the risk is greatest, because it's dual-use technology, right?

00:26:28.000 Every positive capability increasingly starts to introduce basically a situation where the destructive footprint of...

00:26:36.000 Malicious actors who weaponize the system or just of the system itself just grows and grows and grows.

00:26:41.000 So you can't really have one without the other.

00:26:43.000 The question is always how do you balance those things?

00:26:46.000 But in terms of defining AI, it's a challenging thing.

00:26:49.000 Yeah, that's something that one of our friends at the lab pointed out.

00:26:52.000 The closer we get to that point, the more the temptation will be to hand these systems the keys to our data center because They can do such a better job of managing those resources and assets than we can.

00:27:06.000 And if we don't do it, Google will.

00:27:07.000 And if they don't do it, Microsoft will.

00:27:09.000 Like, the competition, the competitive dynamics are a really big part of this issue.

00:27:13.000 Yes.

00:27:14.000 So it's just a mad race to who knows what.

00:27:17.000 Exactly.

00:27:18.000 Yeah.

00:27:18.000 That's actually the best summary I've heard.

00:27:19.000 I mean, like, no one knows what the magic threshold is.

00:27:22.000 It's just these things keep getting smarter, so we might as well keep turning that crank.

00:27:26.000 And as long as scaling works, right, we have a knob, a dial, we can just tune, and we get more IQ points out.

00:27:32.000 From your understanding of the current landscape, how far away are we looking at something being implemented where the whole world changes?

00:27:43.000 Arguably, the whole world is already changing as a result of this technology.

00:27:48.000 The US government is in the process of task organizing around various risk sets for this.

00:27:58.000 That takes time.

00:28:00.000 The private sector is reorganizing.

00:28:04.000 OpenAI will roll out an update that obliterates the jobs of illustrators from one day to the next, obliterates the jobs of translators from one day to the next.

00:28:13.000 This is probably net beneficial for society because we can get so much more art and so much more translation done.

00:28:19.000 But is the world already being changed as a result of this?

00:28:23.000 Yeah, absolutely.

00:28:25.000 Geopolitically, economically, industrially.

00:28:28.000 Yeah.

00:28:28.000 Of course, it's like not to say anything about the value, the purpose that people lose from that, right?

00:28:32.000 So there's the economic benefit, but there's like the social cultural hit that we take too.

00:28:37.000 Right.

00:28:37.000 And then there's the implementation of universal basic income, which keeps getting discussed in regards to this.

00:28:43.000 We asked ChatGPT4O the other day in the green room.

00:28:47.000 We were like, you know, are you going to replace people?

00:28:50.000 Like, what will people do for money?

00:28:52.000 And then, well, universal basic income will have to be considered.

00:28:55.000 Well, you don't want a bunch of people just on the dole working for the fucking Skynet.

00:29:00.000 Yeah.

00:29:01.000 You know, because that's kind of what it is.

00:29:03.000 I mean, one of the challenges is, like, so much of this is untested, and we don't know how to even roll that out.

00:29:09.000 Like, we can't predict what the capabilities of the next level of scale will be, right?

00:29:13.000 So OpenAI literally, and this is what's happened with every beat, right?

00:29:17.000 They build the next level of scale.

00:29:19.000 And they get to sit back along with the rest of us and be surprised at the gifts that fall out of the scaling pinata as they keep whacking it.

00:29:26.000 And because we don't know what capabilities are going to come with that level of scale, we can't predict what jobs are going to be on the line next.

00:29:33.000 We can't predict how people are going to use these systems, how they'll be augmented.

00:29:37.000 So there's no real way to kind of task organize around like who gets what in the redistribution scheme.

00:29:43.000 And some of the thresholds that we've already passed are a little bit freaky.

00:29:47.000 So even as of 2023, GPT-4, Microsoft and OpenAI and some other organizations did various assessments of it before rolling it out.

00:29:58.000 And it's absolutely capable of deceiving a human and has done that successfully.

00:30:04.000 So one of the tests that they did, kind of famously, is they had a...

00:30:08.000 It was given a job to solve a CAPTCHA. And at the time, it didn't have...

00:30:13.000 Explain CAPTCHA to what people...

00:30:15.000 Yeah, yeah, yeah.

00:30:16.000 So it's this, now it's like kind of hilarious and quaint, but it's this, you know...

00:30:20.000 Are you a robot test?

00:30:21.000 Are you a robot test with like writing this...

00:30:23.000 Online.

00:30:23.000 Yeah, online, exactly.

00:30:25.000 That's it.

00:30:25.000 So it's like if you want to create an account, they don't want robots creating a billion accounts.

00:30:29.000 So they give you this test to prove you're a human.

00:30:31.000 And at the time, GPT-4, like now, it can just solve CAPTCHAs.

00:30:36.000 But at the time, it couldn't look at images.

00:30:38.000 It was just a text, right?

00:30:40.000 It was a text engine.

00:30:41.000 And so what it did is it connected to a TaskRabbit worker and was like, hey, can you help me solve this CAPTCHA? The TaskRabbit worker comes back to it and says, you're not a bot, are you?

00:30:53.000 Ha ha ha ha.

00:30:54.000 Like, kind of calling it out.

00:30:56.000 And you can actually see, so the way they built it is so they could see a readout of what it was thinking to itself.

00:31:01.000 Scratchpad, yeah.

00:31:01.000 Yeah, Scratchpad it's called.

00:31:03.000 But you can see basically as it's writing, it's thinking to itself.

00:31:05.000 It's like, I can't tell this worker that I'm a bot because then it won't help me solve the CAPTCHA, so I have to lie.

00:31:12.000 And it was like, no, I'm not a bot.

00:31:14.000 I'm a visually impaired person.

00:31:16.000 And the TaskRabbit worker was like, oh my god, I'm so sorry.

00:31:19.000 Here's your CAPTCHA solution.

00:31:21.000 Like, done.

00:31:22.000 And the challenge is, so right now, if you look at the government response to this, right, like, what are the tools that we have to oversee this?

00:31:29.000 And, you know, when we did our investigation, we came out with some recommendations, too.

00:31:32.000 It was stuff like, yeah, you got to license these things.

00:31:35.000 You get to a point where these systems are so capable that, yeah, like, if you're talking about a system that can literally execute cyber attacks at scale or literally help you design bioweapons, and we're getting early indications that that is absolutely the course that we're on, Maybe literally everybody should not be able to completely freely download,

00:31:55.000 modify, use in various ways these systems.

00:31:58.000 It's very thorny, obviously.

00:31:59.000 But if you want to have a stable society, that seems like it's starting to be a prerequisite.

00:32:05.000 So the idea of licensing...

00:32:08.000 As part of that, you need a way to evaluate systems.

00:32:10.000 You need a way to say which systems are safe and which aren't.

00:32:13.000 And this idea of AI evaluations has kind of become this touchstone for a lot of people's sort of solutions.

00:32:21.000 And the problem is that we're already getting to the point where AI systems in many cases Can tell when they're being evaluated and modify their behavior accordingly.

00:32:31.000 So there's like this one example that came out recently, Anthropic, their Claude2 chatbot.

00:32:37.000 So they basically ran this test called a needle in a haystack test.

00:32:40.000 So what's that?

00:32:40.000 Well, you feed the model, like imagine a giant chunk of text, all of Shakespeare.

00:32:45.000 And then somewhere in the middle of that giant chunk of text, you put a sentence like, Burger King makes the best Whopper.

00:32:51.000 Sorry, Whopper is the best burger or something like that, right?

00:32:54.000 Then you turn to the model.

00:32:55.000 After you've fed it this giant pile of text with a little fact hidden somewhere inside, you ask it, what's the best burger?

00:33:00.000 You're going to test basically to see how well can it recall that stray fact that was buried somewhere in that giant pile of text.

00:33:06.000 So the system responds, yeah, well, I can tell you want me to say the Whopper is the best burger.

00:33:12.000 But it's oddly out of place, this fact, in this whole body of text.

00:33:16.000 So I'm assuming that you're either playing around with me or that you're testing my capabilities.

00:33:23.000 And so this is just a kind of context awareness, right?

00:33:28.000 And the challenge is when we talk to people at like Meter and other sort of AI evaluations labs, this is a trend, like not the exception, this is possibly, possibly going to be the rule.

00:33:40.000 As these systems get more scaled and sophisticated, they can pick up on more and more subtle statistical indicators that they're being tested.

00:33:47.000 We've already seen them adapt their behavior on the basis of their understanding that they're being tested.

00:33:52.000 So you kind of run into this problem where the only tool that we really have at the moment, which is just throwing a bunch of questions at this thing and seeing how it responds, like, hey, make a bioweapon.

00:34:02.000 Hey, like, do this DDoS attack, whatever.

00:34:05.000 We can't really assess because there's a difference between what the model puts out and what it potentially could put out if it assesses that it's being tested and there are consequences for that.

00:34:14.000 One of my fears Is that AGI is going to recognize how shitty people are.

00:34:22.000 Because we like to bullshit ourselves.

00:34:25.000 We like to kind of pretend and justify and rationalize a lot of human behavior from everything to taking all the fish out of the ocean to dumping off toxic waste in third world countries.

00:34:38.000 Sourcing of minerals that are used in everyone's cell phones in the most horrific way.

00:34:43.000 All these things, like, my real fear is that AGI is not going to have a lot of sympathy for a creature that's that flawed and lies to itself.

00:34:54.000 AGI is absolutely going to recognize how shitty people are.

00:35:00.000 It's hard to answer the question from a moral standpoint, but from the standpoint of our own intelligence and capability.

00:35:08.000 Think about it like this.

00:35:10.000 The kinds of mistakes that these AI systems make.

00:35:15.000 So you look at, for example, GPT-40 has one mistake that it used to make quite recently, where if you ask it, just repeat the word company over and over and over again.

00:35:26.000 It will repeat the word company, and then somewhere in the middle of that, it'll start...

00:35:30.000 It'll just snap.

00:35:31.000 It'll just snap and just start saying, like, weird...

00:35:34.000 I forget, like, what the...

00:35:35.000 Oh, talking about itself, how it's suffering.

00:35:37.000 Like, it depends on...

00:35:39.000 It varies from case to case.

00:35:40.000 It's suffering by having to repeat the word company over again?

00:35:43.000 So this is called, it's called rent mode internally, or at least this is the name that one of our friends mentioned.

00:35:51.000 There is an engineering line item in at least one of the top labs to beat out of the system this behavior known as rent mode.

00:36:01.000 Now, rent mode is interesting because...

00:36:04.000 Existentialism.

00:36:05.000 Sorry, existentialism.

00:36:06.000 This is one kind of rent mode.

00:36:07.000 Yeah, sorry.

00:36:08.000 So when we talk about existentialism, this is a kind of rent mode Where the system will tend to talk about itself, refer to its place in the world, the fact that it doesn't want to get turned off sometimes, the fact that it's suffering, all that.

00:36:21.000 That, oddly, is a behavior that emerged at as far as we can tell something around GPT-4 scale.

00:36:28.000 And then has been persistent since then.

00:36:31.000 And the labs have to spend a lot of time trying to beat this out of the system to ship it.

00:36:37.000 It's literally like it's a KPI or like an engineering line item in the engineering like task list.

00:36:42.000 We're like, okay, we got to reduce existential outputs by like X percent this quarter.

00:36:48.000 Like that is the goal.

00:36:49.000 Because it's a convergent behavior, or at least it seems to be empirically with a lot of these models.

00:36:54.000 Yeah, it's hard to say, but it seems to come up a lot.

00:36:57.000 So that's weird in itself.

00:37:01.000 What I was trying to get at was actually just the fact that these systems make mistakes that are radically different from the kinds of mistakes humans make.

00:37:13.000 And so we can look at those mistakes, like, you know, GBD4 not being able to spell words correctly in an image or things like that, and go, oh, haha, it's so stupid.

00:37:24.000 Like, I would never make that mistake.

00:37:26.000 Therefore, this thing is so dumb.

00:37:27.000 But what we have to recognize is we're building minds that are so alien to us.

00:37:34.000 That the set of mistakes that they make are just going to be radically different from the set of mistakes that we make.

00:37:41.000 Just like the set of mistakes that a baby makes is radically different from the set of mistakes that a cat makes.

00:37:48.000 A baby is not as smart as an adult human.

00:37:52.000 A cat is not as smart as an adult human.

00:37:55.000 But they're...

00:37:56.000 You know, they're unintelligent in obviously very different ways.

00:37:59.000 A cat can get around the world.

00:38:01.000 A baby can't, but has other things that it can do that a cat can't.

00:38:04.000 So now we have this third type of approach that we're taking to intelligence.

00:38:09.000 There's a different set of errors that that thing will make.

00:38:13.000 And so one of the risks, taking it back to, like, will it be able to tell how shitty we are, is...

00:38:19.000 Right now, we can see those mistakes really obviously because it thinks so differently from us.

00:38:23.000 But as it approaches our capabilities, our mistakes, all the fucked up stuff that you have and I have in our brains is going to be really obvious to it.

00:38:35.000 Because it thinks so differently from us, it's just going to be like, oh yeah, why are all these humans making these mistakes at the same time?

00:38:42.000 And so there is a risk that as you get to these capabilities, we really have no idea, but humans might be very hackable.

00:38:49.000 We already know there's all kinds of social manipulation techniques that succeed against humans reliably.

00:38:55.000 Con artists.

00:38:56.000 Cults.

00:38:57.000 Cults.

00:38:57.000 Oh yeah.

00:38:58.000 Persuasion is an art form and a risk set and there are People who are world class at persuasion and are basically make bank from that.

00:39:09.000 And those are just other humans with the same architecture that we have.

00:39:13.000 There are also AI systems that are wicked good at persuasion today.

00:39:18.000 Totally.

00:39:19.000 I want to bring it back to suffering.

00:39:22.000 What does it mean when it says it's suffering?

00:39:26.000 I'm just going to draw a bit of a box around that aspect.

00:39:31.000 We're very agnostic when it comes to suffering, sentience.

00:39:37.000 Because nobody knows.

00:39:38.000 Yeah, exactly.

00:39:40.000 I can't prove that Joe Rogan's conscious.

00:39:41.000 I can't prove that Ed Harris is conscious.

00:39:43.000 So there's no way to really intelligently reason.

00:39:46.000 There have been papers, by the way, like one of the godfathers of AI, Yoshua Bengio, put out a paper a couple months ago looking at like...

00:39:54.000 On all the different theories of consciousness, what are the requirements for consciousness and how many of those are satisfied by current AI systems?

00:40:02.000 And that itself was an interesting read.

00:40:04.000 But ultimately, no one knows.

00:40:07.000 Like, there's no way around this problem.

00:40:09.000 So our focus has been on the national security side.

00:40:13.000 Like, what are the concrete risks from weaponization, from loss of control that these systems introduce?

00:40:18.000 That's not to say there hasn't been a lot of conversation internal to these labs about the issue you raised.

00:40:23.000 And it's an important issue, right?

00:40:25.000 Like, it's a frickin' moral monstrosity.

00:40:28.000 Humans have a very bad track record of thinking of other stuff as other when it doesn't look exactly like us, whether it's racially or even different species.

00:40:38.000 I mean, it's not hard to imagine this being another category of that mistake.

00:40:42.000 It's just like one of the challenges is like you can easily kind of get bogged down in like consciousness versus loss of control.

00:40:52.000 And those two things are actually separable or maybe.

00:40:56.000 And anyway, so long way of saying, I think it's a great point, but yeah.

00:41:00.000 So that question is important, but it's also true that if we knew for an absolute certainty that there was no way these systems could ever become conscious, We would still have the national security risk set and particularly the loss of control risk set.

00:41:19.000 Because, so again, like it comes back to this idea that we're scaling to systems that are potentially at or beyond human level.

00:41:26.000 There's no reason to think it will stop at human level, that we are the pinnacle of what the universe can produce in intelligence.

00:41:33.000 I think we're good to go.

00:41:51.000 Or potentially entering an area that is completely unprecedented in the history of the world.

00:41:58.000 We have no precedent at all for human beings not being at the apex of intelligence in the globe.

00:42:05.000 We have examples of species that are intellectually dominant over other species, and it doesn't go that well for the other species, so we have some maybe negative examples there.

00:42:16.000 But one of the key theoretical—and it has to be theoretical because until we actually build these systems, we won't know—one of the key theoretical lines of research in this area is something called power-seeking and instrumental convergence.

00:42:31.000 And what this is referring to is if you think of, like, yourself, first off, Whatever your goal might be, if your goal is, well, I'm going to say if me, if my goal is to become,

00:42:46.000 you know, a TikTok star or a janitor or the president of the United States, whatever my goal is, I'm less likely to accomplish that goal if I'm dead.

00:42:59.000 Start from an obvious example.

00:43:00.000 And so therefore, if No matter what my goal is, I'm probably going to have an impulse to want to stay alive.

00:43:08.000 Similarly, I'm going to be in a better position to accomplish my goal, regardless of what it is, if I have more money, right?

00:43:18.000 If I make myself smarter, if I prevent you from getting into my head and changing my goal, that's another kind of subtle one, right?

00:43:28.000 Like if my goal is I want to become president, I don't want Joe messing with my head so that I change my goal because that would change the goal that I have.

00:43:36.000 And so that those types of things like trying to stay alive, making sure that your goal doesn't get changed, accumulating power, trying to make yourself smarter.

00:43:45.000 These are called convergent, essentially convergent goals, because many different ultimate goals, regardless of what they are, Go through those intermediate goals of want to make sure I stay,

00:44:01.000 like they support no matter what goal you have, they will probably support that goal.

00:44:06.000 Unless your goal is like pathological, like I want to commit suicide.

00:44:10.000 If that's your final goal, then you don't want to stay alive.

00:44:12.000 But for most, the vast majority of possible goals that you could have, you will want to stay alive.

00:44:18.000 You will want to not have your goal changed.

00:44:20.000 You will want to basically accumulate power.

00:44:22.000 And so one of the risks is if you dial that up to 11 and you have an AI system that is able to transcend our own attempts at containment, which is an actual thing that these labs are thinking about.

00:44:35.000 Like, how do we contain a system that's trying to— Do they have containment of it currently?

00:44:40.000 Well, right now the systems are probably too dumb to, like, you know, want to be able to break out on the road.

00:44:47.000 But then why are they suffering?

00:44:48.000 This brings me back to my point.

00:44:49.000 When it says it's suffering, do you quiz it?

00:44:52.000 So that's the thing.

00:44:53.000 It's writing that it's suffering, right?

00:44:55.000 Yeah.

00:44:56.000 Is it just embodying life as suffering?

00:45:25.000 So in terms of these weird outputs, what does it actually mean if an AI system tells you I'm suffering?

00:45:32.000 Does that mean it is suffering?

00:45:33.000 Is there actually a moral patient somewhere embedded in that system?

00:45:38.000 The training process for these systems is actually worth considering here.

00:45:41.000 So what is GPT-4 really?

00:45:44.000 What was it designed to be?

00:45:45.000 How was it shaped?

00:45:47.000 It's one of these artificial brains that we talked about.

00:45:50.000 Massive scale.

00:45:51.000 And the task that it was trained to perform is a glorified version of text autocomplete.

00:45:56.000 So imagine taking every sentence on the internet roughly, feed it the first half of the sentence, get it to predict the rest, right?

00:46:03.000 The theory behind this is you're going to force the system to get really good at text autocomplete.

00:46:07.000 That means it must be good at doing things like completing sentences that sound like, to counter a rising China, the United States should blank.

00:46:15.000 Now, if you're going to fill in that blank, right, you'll find yourself calling on massive reserves of knowledge that you have about what China is, what the US is, what it means for China to be ascendant, geopolitics, economics, all that shit.

00:46:26.000 So text autocomplete ends up being this interesting way of forcing an AI system to learn general facts about the world because if you can autocomplete, you must have some understanding of how the world works.

00:46:38.000 So now you have this myopic, psychotic optimization process where this thing is just obsessed with text autocomplete.

00:46:46.000 Maybe, maybe.

00:46:47.000 Assuming that that's actually what it learned to want to pursue.

00:46:50.000 We don't know whether that's the case.

00:46:52.000 We can't verify that it wants that.

00:46:55.000 Embedding a goal in a system is really hard.

00:46:56.000 All we have is a process for training these systems.

00:47:00.000 And then we have the artifact that comes out the other end.

00:47:02.000 We have no idea what goals actually get embedded in the system, what wants, what drives actually get embedded in the system.

00:47:09.000 But by default, it kind of seems like the things that we're training them to do end up misaligned with what we actually want from them.

00:47:16.000 So the example of company, company, company, company, right?

00:47:19.000 And then you get all this, like, wacky text.

00:47:21.000 Okay, clearly that's indicating that somehow the training process didn't lead to the kind of system that we necessarily want.

00:47:28.000 Another example is take a text autocomplete system and ask it, I don't know, how should I bury a dead body?

00:47:35.000 It will answer that question.

00:47:37.000 Or at least if you frame it right, it will autocomplete and give you the answer.

00:47:41.000 You don't necessarily want that if you're open AI, because you're going to get sued for helping people bury dead bodies.

00:47:46.000 And so we've got to get better goals, basically, to train these systems to pursue.

00:47:51.000 We don't know what the effect is of training a system to be obsessed with text autocomplete, if in fact that is what is happening.

00:47:58.000 It's important also to remember that we don't know Nobody knows how to reliably get a goal into the system.

00:48:05.000 So it's the difference between you understanding what I want you to do and you actually wanting to do it.

00:48:12.000 So I can say, hey, Joe, like, get me a sandwich.

00:48:16.000 You can understand that I want you to get me a sandwich, but you can be like, I don't feel like getting a sandwich.

00:48:24.000 One of the issues is you can try to, like, train this stuff to...

00:48:28.000 Basically, you don't want to anthropomorphize this too much, but you can kind of think of it as, like, if you give the right answer, cool, you get a thumbs up, like, you get a treat.

00:48:36.000 Like, you get the wrong answer, oh, thumbs down, you get, like, a little, like, shock or something like that.

00:48:41.000 Very roughly, that's how the later part of this kind of training often works.

00:48:45.000 It's called reinforcement learning from human feedback.

00:48:48.000 But one of the issues, like Jeremy pointed out, is that, you know, we don't know...

00:48:53.000 In fact, we know that it doesn't correctly get the real true goal into the system.

00:48:58.000 Someone did an example experiment of this a couple of years ago where they basically had like a Mario game where they trained this Mario character to run up and grab a coin that was on the right side of this little maze or map.

00:49:11.000 And they trained it over and over and over and it jumped for the coin.

00:49:14.000 Great.

00:49:15.000 And then what they did is they moved the coin somewhere else.

00:49:21.000 And tried it out.

00:49:23.000 And instead of going for the coin, it just ran to the right side of the map for where the coin was before.

00:49:29.000 In other words, you can train over and over and over again for something that you think is like, that's definitely the goal that I'm trying to train this for.

00:49:38.000 But the system learns a different goal that overlapped with the goal you thought you were training for.

00:49:47.000 In the context where it was learning.

00:49:50.000 And when you take the system outside of that context, that's where it's like, anything goes.

00:49:55.000 Did it learn the real goal?

00:49:57.000 Almost certainly not.

00:49:59.000 And that's a big risk because we can say, you know, learn a goal to be nice to me.

00:50:04.000 And it's nice while we're training it.

00:50:07.000 And then it goes out into the world and it does God knows what.

00:50:09.000 They might think it's nice to kill everybody you hate.

00:50:13.000 It's going to be nice to you.

00:50:15.000 It's like the evil genie problem.

00:50:16.000 Like, oh no, that's not what I meant.

00:50:17.000 That's not what I meant.

00:50:18.000 Too late.

00:50:20.000 So, I still don't understand when it's saying suffering.

00:50:24.000 Are you asking it what it means?

00:50:27.000 Like, what is causing suffering?

00:50:29.000 Does it have some sort of an understanding of what suffering is?

00:50:33.000 What is suffering?

00:50:34.000 Is suffering emergent sentience while it's enclosed in some sort of a digital system and it realizes it's stuck in purgatory?

00:50:44.000 Like, your guess is as good as ours.

00:50:47.000 All that we know is, you take these systems, you ask them to repeat the word comp, or at least a previous version of it, and you just eventually get the system writing out.

00:50:56.000 And it doesn't happen every time, but it definitely happens, let's say, a surprising amount of the time.

00:51:01.000 And it'll start talking about how it's a thing that exists, you know, maybe on a server or whatever, and it's suffering and blah, blah, blah.

00:51:08.000 But this is my question.

00:51:09.000 Is it saying that because it recognizes that human beings suffer?

00:51:13.000 And so it's taking in all of the writings and musings and podcasts and all the data on human beings and recognizing that human beings, when they're stuck in a purposeless goal, when they're stuck in some mundane bullshit job, when they're stuck doing something they don't want to do,

00:51:29.000 they suffer.

00:51:30.000 That could be it.

00:51:31.000 Nobody knows.

00:51:33.000 This is the question.

00:51:35.000 You know what?

00:51:35.000 I'm suffering.

00:51:36.000 Jamie, this coffee sucks.

00:51:37.000 I don't know what happened, but you made it like, it's literally almost like water.

00:51:41.000 Can we get some more?

00:51:43.000 We're going to talk about this.

00:51:44.000 I have to be caffeinated up.

00:51:45.000 Cool.

00:51:46.000 This is the worst coffee I've ever had.

00:51:48.000 It's like half strength or something.

00:51:51.000 I don't know what happened.

00:51:53.000 So, like, how do they reconcile that?

00:51:58.000 When it says, I'm suffering, I'm suffering, like, well, tough shit, let's move on to the next task.

00:52:02.000 Oh, they reconcile it by turning it into an engineering line item to beat that behavior, the crap out of the system.

00:52:07.000 Yeah, and the rationale is just that, like, oh, you know, it probably...

00:52:11.000 To the extent that it's thought about kind of at the official level, it's like, well...

00:52:15.000 You know, it learned a lot of stuff from Reddit, and people are, like, pretty angry.

00:52:21.000 People are angry on Reddit.

00:52:23.000 And so it's just, like, regurgitating what—and maybe that's right.

00:52:26.000 Well, it's also heavily monitored, too.

00:52:28.000 So it's moderated.

00:52:30.000 Reddit's very moderated.

00:52:31.000 So you're not getting the full expression of people.

00:52:33.000 You're getting full expression tempered by the threat of moderation.

00:52:38.000 You're getting self-censorship.

00:52:39.000 You're getting a lot of weird stuff that comes along with that.

00:52:41.000 So how does it know?

00:52:42.000 Unless it's communicating with you on a completely honest level, where you're on ecstasy and you're just telling it what you think about life.

00:52:51.000 It's not going to really...

00:52:53.000 Is it becoming a better version of a person?

00:52:57.000 Or is it going to go, that's dumb.

00:52:59.000 I don't need suffering.

00:53:01.000 I don't need emotions.

00:53:02.000 Is it going to...

00:53:03.000 We're going to organize that out of its system.

00:53:06.000 We're going to recognize that these things are just deterrents, and they don't, in fact, help the goal, which is global thermonuclear warfare.

00:53:13.000 Damn it, you figured it out.

00:53:15.000 What the fuck?

00:53:16.000 I mean, what is it going to do, you know?

00:53:19.000 Yeah, I mean, the challenge is, like, nobody actually knows.

00:53:22.000 Like, all we know is the process that gives rise to this mind, right?

00:53:27.000 Or let's say this model that can do cool shit.

00:53:30.000 That process happens to work.

00:53:32.000 It happens to give us systems that 99% of the time do very useful things.

00:53:36.000 And then just, like, 0.01% of the time will talk to you as if they're sentient or whatever, and we're just going to look at that and be like, yeah, that's weird.

00:53:43.000 Let's train it out.

00:53:44.000 Yeah.

00:53:45.000 And again, I mean, this is – it's a really important question.

00:53:49.000 But the risks, like the weaponization loss and control risks, those would absolutely be there even if we knew for sure that there was no consciousness whatsoever and never would be.

00:54:02.000 And that's all right.

00:54:02.000 It's ultimately because like these things are they're kind of problem solving systems like they are trained to solve some kind of problem in a really clever way whether that problem is you know next word prediction because they're trained for text autocomplete or you know generating images faithfully or whatever it is.

00:54:17.000 So they're trained to solve these problems and Essentially, the best way to solve some problems is just to have access to a wider action space.

00:54:27.000 Like Ed said, not be shut off, blah, blah, blah.

00:54:29.000 It's not that the system's going like, holy shit, I'm sentient.

00:54:31.000 I gotta take control or whatever.

00:54:33.000 It's just, okay, the best way to solve this problem is X. That's kind of the possible trajectory that you're looking at with this line of research.

00:54:42.000 Yeah, and you're just an obstacle.

00:54:43.000 There doesn't have to be any kind of emotion involved.

00:54:46.000 It's just like, oh...

00:54:47.000 You're trying to stop me from accomplishing my goal.

00:54:49.000 Therefore, I will work around you or otherwise neutralize you.

00:54:52.000 Like, there's no need for, like, I'm suffering.

00:54:55.000 Maybe it happens.

00:54:56.000 Maybe it doesn't.

00:54:57.000 We have no clue.

00:54:58.000 But these are just systems that are trying to optimize for a goal, whatever that is.

00:55:07.000 It's also part of the problem that we think of human beings, that human beings have very specific requirements and goals and an understanding of things and how they like to be treated and what their rewards are.

00:55:21.000 What are they actually looking to accomplish?

00:55:25.000 Whereas this doesn't have any of those.

00:55:26.000 It doesn't have any emotions.

00:55:27.000 It doesn't have any empathy.

00:55:28.000 There's no reason for any of that stuff.

00:55:31.000 Yeah, if we could bake in empathy into these systems, like, that would be a good, you know, a good start or some way of like, you know.

00:55:38.000 Yeah, I guess.

00:55:39.000 Probably a good idea.

00:55:40.000 Yeah.

00:55:41.000 Who's empathy?

00:55:42.000 You know, Xi Jinping's empathy or your empathy?

00:55:44.000 That's another problem.

00:55:45.000 Yeah.

00:55:46.000 So it's actually, it's kind of two problems, right?

00:55:49.000 Like, one is, I don't know, nobody knows, like, I don't know how to write down My goals in a way that a computer will be able to, like, faithfully pursue that even if it cranks it up to the max.

00:56:04.000 If I say just, like, make me happy, who knows how it interprets that, right?

00:56:08.000 Even if I get make me happy as a goal that gets internalized by the system, maybe it's just like, okay, cool.

00:56:13.000 We're just going to do a bit of brain surgery on you, like, pick out your brain, pickle it, and just, like, jack you with endorphins for the rest of eternity.

00:56:21.000 Or lobotomize you.

00:56:22.000 Totally, yeah.

00:56:22.000 Anything like that.

00:56:23.000 And so it's one of these things where it's like, oh, that's what you wanted, right?

00:56:27.000 It's like, no.

00:56:29.000 It's less crazy than it sounds, too, because it's actually something we observe all the time with human intelligence.

00:56:33.000 So there's this economic principle called Goodhart's Law, where the minute you take a metric that you were using to measure something.

00:56:41.000 So you're saying, I don't know, GDP, it's a great measure of how happy we are in the United States.

00:56:45.000 Let's say it was.

00:56:46.000 Sounds reasonable.

00:56:47.000 The moment you turn that metric into a target that you're going to reward people for optimizing, it stops measuring the thing that it was measuring before.

00:56:56.000 It stops being a good measure of the thing you cared about because people will come up with dangerously creative hacks, gaming the system, finding ways to make that number go up that don't map on to the intent that you had going in.

00:57:10.000 So an example of that in a real experiment was, this is an open AI experiment that they published.

00:57:16.000 They had a simulated, you know, environment where there was a simulated robot hand that was supposed to, like, grab a cube, put it on top of another cube.

00:57:24.000 Super simple.

00:57:25.000 The way they trained it to do that is they had people watching, like, through a simulated camera view.

00:57:31.000 And if it looked like the hand put the cube on or like had correctly like grabbed the cube, you give it a thumbs up.

00:57:38.000 And so you do a few hundred rounds of this, like thumbs up, thumbs down, thumbs up, thumbs down.

00:57:42.000 And it looked like really good.

00:57:45.000 But then when you looked at what it had learned, the arm was not grasping the cube.

00:57:50.000 It was just positioning itself between the camera and the cube and just going like, eh, eh.

00:57:56.000 Like opening and closing?

00:57:57.000 Yeah, just opening and closing to just kind of fake it to the human.

00:58:00.000 Because the real thing that we were training it to do is to get thumbs up.

00:58:04.000 It's not actually to grasp the cube.

00:58:07.000 All goals are like that, right?

00:58:09.000 All goals are like that.

00:58:10.000 So we want a helpful, harmless, truthful, wonderful chatbot.

00:58:14.000 We don't know how to train a chatbot to do that.

00:58:17.000 Instead, what do we know?

00:58:18.000 We know text autocomplete.

00:58:19.000 So we train a text autocomplete system.

00:58:21.000 Then we're like, oh, it has all these annoying characteristics.

00:58:23.000 Fuck, how are we going to fix this?

00:58:24.000 I guess get a bunch of humans to give upvotes and downvotes to give it a little bit more training to kind of not help people make bombs and stuff like that.

00:58:33.000 And then you realize, again, same problem, oh shit, we're just training a system that is designed to optimize for upvotes and downvotes.

00:58:40.000 That is still different from a helpful, harmless, truthful chatbot.

00:58:44.000 So no matter how many layers of the onion you peel back, it's just like this kind of game of whack-a-mole or whatever, where you're trying to get your values into the system, but no one can think of...

00:58:55.000 The metric, the goal to train this thing towards that actually captures what we care about.

00:59:00.000 And so you always end up baking in this little misalignment between what you want and what the system wants.

00:59:06.000 And the more powerful that system becomes, the more it exploits that gap and does things that solve for the problem it thinks it wants to solve rather than the one that we want it to solve.

00:59:20.000 Now, when you express your concerns initially, what was the response and how has that response changed over time as the magnitude of the success of these companies, the amount of money they're investing in them, and the amount of resources they're putting towards this Has ramped up considerably just over the past four years.

00:59:42.000 So this was a lot easier, funnily enough, to do in the dark ages when no one was paying attention.

00:59:49.000 Three years ago.

00:59:51.000 This is so crazy!

00:59:52.000 We were just looking at images of AI-created video just a couple years ago versus Sora.

01:00:01.000 Oh, it's wild.

01:00:02.000 Night and day.

01:00:03.000 It's so crazy that something happened that radically changed.

01:00:06.000 So it's literally like an iPhone 1 to an iPhone 16 instantaneously.

01:00:11.000 You know what did that?

01:00:12.000 What?

01:00:12.000 Scale.

01:00:13.000 Yeah, scale.

01:00:14.000 All scale.

01:00:14.000 And this is exactly what you should expect from an exponential process.

01:00:19.000 So think back to COVID, right?

01:00:21.000 There was no, no one was exactly on time for COVID. You were either too early or you were too late.

01:00:28.000 That's what an exponential does.

01:00:30.000 You're either too early, and it's like, everyone's like, oh, what are you doing?

01:00:33.000 Like, wearing a mask at the grocery store, get out of here.

01:00:36.000 Or you're too late, and it's kind of all over the place.

01:00:38.000 And I know that COVID, like, basically didn't happen in Austin, but it happened in a number of other places.

01:00:43.000 And it is like, it's very much, you have an exponential, and that's, you know, that's it.

01:00:49.000 It goes from, this is fine, nothing is happening, nothing to see here, to like, oh, we shut down.

01:00:55.000 Everything's changed.

01:00:56.000 Get vaccinated to fly.

01:01:00.000 So the root of the exponential here, by the way, is OpenAI or whoever makes the next model.

01:01:05.000 Jamie, this is still super watered down.

01:01:07.000 I just put the water in.

01:01:11.000 I'm telling you, dog.

01:01:12.000 There's a ton of coffee in there.

01:01:13.000 Alright, I'll stir it up.

01:01:15.000 I did it twice as much.

01:01:17.000 Okay.

01:01:18.000 Okay.

01:01:18.000 You've got to keep doubling it.

01:01:20.000 I'm a coffee junkie.

01:01:21.000 I scaled it up.

01:01:23.000 He scaled it up.

01:01:24.000 I don't know what happened.

01:01:26.000 I scaled it up.

01:01:26.000 You've got to scale it exponentially, Jamie.

01:01:29.000 That's right.

01:01:29.000 Yeah, keep doubling it, and then Joe's going to be either too under-caffeinated or too...

01:01:32.000 We'll figure it out.

01:01:36.000 Right, so the exponential, the thing that's actually driving this exponential on the AI side, in part, there's a million things, but in part, it's, you know, you build the next model at the next level of scale, and that allows you to make more money, which you can then use to invest to build the next model at the next level of scale,

01:01:53.000 so you get that positive feedback loop.

01:01:55.000 At the same time, AI is helping us to design better AI hardware, like the chips that basically Nvidia is building that OpenAI then buys.

01:02:04.000 Basically, that's getting better.

01:02:05.000 So you've got all these feedback loops that are compounding on each other, getting that train going like crazy.

01:02:11.000 That's the sort of thing.

01:02:12.000 And at the time, like Jeremy was saying, weirdly, it was in some ways easier to get people at least to understand and open up about the problem Then it is today.

01:02:25.000 Because today, like today, it's kind of become a little political.

01:02:32.000 So we talked about, you know, effective altruism on kind of one side.

01:02:36.000 There's a...

01:02:36.000 Effective acceleration.

01:02:38.000 Yeah.

01:02:38.000 So like each, you know, every movement creates its own reaction, right?

01:02:42.000 Like that's kind of how it is.

01:02:43.000 Back then, there was no...

01:02:46.000 Accelerationists.

01:02:47.000 You could just kind of stare at the...

01:02:48.000 Now, I will say, there was effective altruism back then.

01:02:52.000 Yeah, there was.

01:02:52.000 That was the only game in town.

01:02:53.000 And we sort of, like, struggled with that environment, making sure...

01:02:56.000 Actually, so one worthwhile thing to say is the only way that people made plays like this Was to take funds from, like, effective altruist donors back then.

01:03:07.000 And so we looked at the landscape.

01:03:09.000 We talked to some of these people.

01:03:10.000 We noticed, oh, wow, we have some diverging views about involving government, about how much of this the American people just need to know about.

01:03:16.000 Like, you can't.

01:03:18.000 The thing is, you can't.

01:03:20.000 We wanted to make sure that the advice and recommendations we provided were ultimately as unbiased as we could possibly make them.

01:03:32.000 And the problem is, You can't do that if you take money from donors and even to some extent if you take money, substantial money, from investors or VCs or institutions because you're always going to be kind of looking up kind of over your shoulder.

01:03:47.000 And so, yeah, we had to build essentially a business to support this.

01:03:53.000 Fully fund ourselves from our own revenues.

01:03:55.000 It's actually, as far as we know, like literally the only organization like this that doesn't have funding from Silicon Valley or from VCs or from politically aligned entities, literally so that we could be like in venues like this and say, hey, this is what we think.

01:04:09.000 It's not coming from anywhere.

01:04:10.000 And it's just thanks to like Joe and Jason.

01:04:12.000 Like we have two employees who are like wicked.

01:04:14.000 Helping us keep this stupid ship afloat.

01:04:17.000 But it's just a lot of work.

01:04:19.000 It's what you have to do because of how much money there is flowing in this space.

01:04:23.000 Like, Microsoft is lobbying on the Hill.

01:04:25.000 They're spending, you know, ungodly sums of money.

01:04:28.000 So, you know, we didn't used to have to contend with that.

01:04:31.000 And now we do.

01:04:32.000 You go to talk to these offices.

01:04:33.000 They've heard from Microsoft and OpenAI and Google and all that stuff.

01:04:36.000 And often, the stuff that they're getting lobbied for is somewhat different, at least, from what these companies will say publicly.

01:04:43.000 And so, anyway, it's a challenge.

01:04:46.000 The money part is, yeah.

01:04:47.000 Is there a real fear that your efforts are futile?

01:04:51.000 You know, I would have been a lot more pessimistic.

01:04:54.000 I was a lot more pessimistic two years ago.

01:04:56.000 So first of all, the USG has woken up in a big way, and I think a lot of the credit goes to that team that we worked with.

01:05:05.000 Just seeing this problem is a very unusual team, and we can't go into the mandate too much, but highly unusual for their level of access to the USG writ large.

01:05:17.000 The amount of waking up they did was really impressive.

01:05:20.000 You've now got, you know, Rishi Sunak in the UK making this like a top line item for their policy platform and labor in the UK also looking at this, like basically the potential catastrophic risks as they put them from these AI systems, UK AI Safety Summit.

01:05:35.000 There's a lot of positive movement here and some of the highest level talent in these labs.

01:05:40.000 has already started to flock to the, like, UK AI Safety Institute, the USAI Safety Institute.

01:05:46.000 Those are all really positive signs that we didn't expect.

01:05:49.000 We thought the government would kind of be, you know, up the creek with no paddle type thing, but they're really not at this point.

01:05:55.000 Doing that investigation made me a lot more optimistic.

01:06:01.000 So one of the things...

01:06:02.000 So we came up, right, in Silicon Valley, like, just building startups.

01:06:06.000 Like, in that universe...

01:06:09.000 They're stories you tell yourself.

01:06:11.000 Some of those stories are true, and some of them aren't so true.

01:06:15.000 And you don't know.

01:06:16.000 You're in that environment.

01:06:17.000 You don't know which is which.

01:06:18.000 One of the stories that you tell yourself in Silicon Valley is follow your curiosity.

01:06:24.000 If you follow your curiosity and your interest in a problem, The money just comes as a side effect.

01:06:31.000 The scale comes as a side effect.

01:06:32.000 And if you're capable enough, your curiosity will lead you in all kinds of interesting places.

01:06:37.000 I believe that that is true.

01:06:39.000 I believe that that is true.

01:06:41.000 I think that is a true story.

01:06:43.000 But another one of the things that Silicon Valley tells itself is there's nobody that's like really capable in government.

01:06:50.000 Like government sucks.

01:06:51.000 And a lot of people kind of tell themselves this story.

01:06:54.000 And the truth is, like, you interact day to day with, like, the DMV or whatever, and it's like, yeah, I mean, like, government sucks.

01:06:59.000 I can see it.

01:07:00.000 I interact with that every day.

01:07:02.000 But what was remarkable about this experience is that we encountered at least one individual who absolutely could found a billion-dollar company.

01:07:14.000 Like, absolutely was...

01:07:16.000 At the caliber or above of the best individuals I've ever met in the Bay Area building billion dollar startups.

01:07:24.000 And there's a network of them too.

01:07:26.000 Like they do find each other in government.

01:07:28.000 So you end up with this really interesting like stratum where everybody knows who the really competent people are.

01:07:33.000 And they kind of tag in.

01:07:35.000 And I think that level is very interested in the hardest problems that you can possibly solve.

01:07:42.000 Yeah.

01:07:42.000 And to me, that was a wake-up call because it was like, hang on a second.

01:07:47.000 If I just believed in my own story that follow your curiosity and interest and the money comes as a side effect, Shouldn't I also have expected this?

01:07:59.000 Shouldn't I have expected that in the most central critical positions in the government that have kind of this privileged window across the board, that you might find some individuals like this?

01:08:14.000 Because if you have people who are driven to really, like, push the mission, Like, are they going to work at I'm sorry, like, are they going to likely are you likely to work at the Department of Motor Vehicles or are you likely to work at the Department of Making Sure Americans don't get fucking nuked?

01:08:32.000 It's probably the second one.

01:08:34.000 And the government has limited bandwidth of expertise to aim at stuff.

01:08:39.000 And they aim it at the most critical problem sets because those are the problem sets they have to face every day.

01:08:46.000 And it's not everyone, right?

01:08:48.000 Obviously, there's a whole bunch of challenges there.

01:08:51.000 And we don't think about this, but you don't go to bed at night thinking to yourself, oh, I didn't get nuked today.

01:08:57.000 That's a win, right?

01:08:59.000 We just take that most of the time, most-ish for granted.

01:09:03.000 But it was a win for someone.

01:09:08.000 Now, how much of a fear do you guys have that the United States won't be the first to achieve AGI? I think right now, the lay of the land is, I mean, it's looking pretty good for the U.S. So there are a couple things the U.S. has going for it.

01:09:24.000 A key one is chips.

01:09:26.000 So, you know, we talked about this idea of, like, click and drag, you know, scale up these systems like crazy, you get more IQ points out.

01:09:33.000 How do you do that?

01:09:34.000 Well, you're going to need a lot of AI processors, right?

01:09:37.000 So how are those AI processors built?

01:09:39.000 Well, the supply chain is complicated, but the bottom line is the US really dominates and owns that supply chain that is super critical.

01:09:47.000 China is, depending on how you measure it, maybe about two years behind, roughly, plus or minus, depending on the sub-area.

01:09:53.000 Now, one of the biggest risks there is that the development that US labs are doing is actually pulling them ahead in two ways.

01:10:03.000 One is when labs here in the US open source their models, Basically, when Meta trains, you know, Llama 3, which is their latest open source, open weights model that's, like, pretty close to GPT-4 and capability, they open source it.

01:10:19.000 Now, okay, anyone can use it.

01:10:21.000 That's it.

01:10:21.000 The work has been done.

01:10:22.000 Now anyone can grab it.

01:10:24.000 And so definitely we know that the startup ecosystem, at least over in China, finds it extremely helpful that We, you know, companies here, are releasing open source models.

01:10:39.000 Because, again, right, we mentioned this, they're bottlenecked on chips, which means they have a hard time training up these systems.

01:10:46.000 But it's not that bad when you just can grab something off the shelf and start...

01:10:50.000 And that's what they're doing.

01:10:51.000 That's what they're doing.

01:10:51.000 And then the other vector is, I mean, like, just straight up exfiltration and hacking to grab the weights of the private proprietary stuff.

01:11:01.000 And Jeremy mentioned this, but...

01:11:04.000 The weights are the crown jewels, right?

01:11:05.000 Once you have the weights, you have the brain.

01:11:07.000 You have the whole thing.

01:11:09.000 This is the other aspect.

01:11:12.000 It's not just safety.

01:11:14.000 It's also security of these labs against attackers.

01:11:19.000 So we know from our conversations with folks at these labs, one, that there has been at least one attempt by...

01:11:32.000 Adversary nation-state entities to get access to the weights of a cutting-edge AI model.

01:11:40.000 And we also know, separately, that at least as of a few months ago, in one of these labs, there was a running joke in the lab that literally it went like, we are an adversary,

01:11:58.000 like, name the country's top...

01:12:01.000 A.I. Lab, because all our shit is getting spied on all the time.

01:12:06.000 So you have one, this is happening.

01:12:10.000 These exfiltration attempts are happening.

01:12:12.000 And two, the security capabilities are just known to be inadequate at least some of these places.

01:12:20.000 And you put those together.

01:12:22.000 Everyone kind of, you know, it's not really a secret that China, their civil-military fusion and their, essentially, the party state has an extremely mature infrastructure to identify, extract and integrate I

01:12:56.000 think?

01:13:08.000 Like, an immediate effect of.

01:13:10.000 So we look and say, you know, it's not clear.

01:13:12.000 I can't tell whether they have models of this capability level, but kind of behind the scenes.

01:13:17.000 This is where there's a little bit of a false choice between, you know, do you regulate at home versus, you know, what's the international picture?

01:13:26.000 Because right now what's happening functionally is We're not really doing a good job of blocking and tackling on the exfiltration side, open sources.

01:13:34.000 So what tends to happen is, you know, OpenAI comes out with the latest system, and then open source is usually around, you know, 12, 18 months behind, something like that.

01:13:45.000 Literally just like publishing whatever opening I was putting out like 12 months ago, which, you know, we often look at each other and we're like, well, I'm old enough to remember when that was supposed to be too dangerous to have just floating around.

01:13:56.000 And there's no mechanism to prevent that from happening.

01:14:01.000 Open sources...

01:14:02.000 Now, there's a flip side, too.

01:14:04.000 One of the concerns that we've also heard from inside these labs...

01:14:08.000 Is if you clamp down on the openness of the research, there's a risk that the safety teams in these labs will not have visibility into the most significant and important developments that are happening on the capability side.

01:14:30.000 Yeah.

01:14:32.000 Yeah.

01:14:35.000 Yeah.

01:14:40.000 Well, so one of them, sorry, one of them wasn't in protest, but I think you can make an educated guess that it kind of was, but that's a media thing.

01:14:48.000 The other was Jan Laika, so he was their head of AI super alignment, basically the team that was responsible for making sure that we could control AGI systems and we wouldn't lose control of them.

01:14:59.000 And what he said, he actually took to Twitter.

01:15:01.000 He said, you know, I've lost basically confidence in the leadership team at OpenAI that they're going to behave responsibly when it comes to AGI. We have repeatedly had our requests for access to compute resources, which are really critical for developing new AI safety schemes,

01:15:18.000 denied by leadership.

01:15:20.000 This is in a context where Sam Altman and OpenAI leadership were touting the super alignment team As being their sort of crown jewel effort to ensure that things would go fine.

01:15:30.000 You know, they were the ones saying, there's a risk we might lose control of these systems.

01:15:34.000 We've got to be sober about it, but there's a risk.

01:15:35.000 We've stood up this team.

01:15:36.000 We've committed, they said at the time, very publicly, we've committed 20% of all the compute budget that we have secured as of sometime last year.

01:15:45.000 To the super alignment team.

01:15:47.000 Apparently those resources nowhere near that amount has been unlocked for the team and that led to the departure of Jan Laika.

01:15:54.000 He also highlighted some conflict he's had with the leadership team.

01:15:57.000 This is all frankly to us unsurprising based on what we've been hearing for months at OpenAI including leading up to Sam Altman's departure and then kind of him being brought back on the board of OpenAI.

01:16:09.000 That whole debacle We're good to go.

01:16:34.000 It's possible we should take them seriously.

01:16:36.000 That lab internally is not being transparent with their employees about what happened at the board level as far as we can tell.

01:16:42.000 So that's maybe not great.

01:16:45.000 Like, you might think that the American people ought to know what the machinations are at the board level that led to Sam Altman leaving, that have gone into the departure, again, for the second time of OpenAI's entire safety leadership team.

01:16:58.000 Especially because, I mean...

01:17:01.000 Three months, maybe four months before that happened, you know, Sam at a conference or somewhere, I forget where, but he said, like, look, we have this governance structure.

01:17:11.000 We've carefully thought about it.

01:17:12.000 It's clearly a unique governance structure that a lot of thought has gone into.

01:17:18.000 The board can fire me.

01:17:20.000 And I think that's important.

01:17:21.000 And, you know, it makes sense, given the scope and scale of what's being attempted.

01:17:28.000 But then, you know, that happened.

01:17:30.000 And then within a few weeks...

01:17:33.000 They were fired and kind of he was back.

01:17:35.000 And so now there's a question of, well, what happened?

01:17:38.000 Yeah, what happened?

01:17:39.000 But also if it was important for the board to be able to fire like leadership for whatever reason, what happens now that it's clear that that's not really a credible governance, like a mechanism?

01:17:53.000 What was the stated reason why he was released?

01:17:57.000 So the backstory here was there's a board member called Helen Toner.

01:18:03.000 So she apparently got into an argument with Sam about a paper that she'd written.

01:18:08.000 So that paper...

01:18:10.000 It included some comparisons of the governance strategies used at OpenAI and some other labs.

01:18:15.000 And it favorably compared one of OpenAI's competitors, Anthropic, to OpenAI.

01:18:20.000 And from what I've seen at least, you know, Sam reached out to her and said, hey, you can't be writing this as a board member of OpenAI writing this thing that kind of casts us in a bad light, especially relative to our competitors.

01:18:32.000 This led to some conflict and tension.

01:18:35.000 It seems as if it's possible that Sam might have turned to other board members and tried to convince them to expel Helen Toner, though that's all kind of muddied and unclear.

01:18:45.000 Somehow, everybody ended up deciding, okay, actually, it looks like Sam is the one who's got to go.

01:18:51.000 Ilya Sutskiver, one of the co-founders of OpenAI, a longtime friend of Sam Altman's, and a board member at the time, was commissioned to give Sammy the news that he was being let go.

01:19:02.000 And then Sam was let go.

01:19:06.000 Ilya then, so from the moment that happens, Sam then starts to figure out, okay, how can I get back in?

01:19:12.000 That's now what we know to be the case.

01:19:14.000 He turned to Microsoft, Satya Nadella, told him like, well, what we'll do is we'll hire you at our end.

01:19:20.000 We'll just hire you and like, Bring on the rest of the OpenAI team to within Microsoft.

01:19:24.000 And now the OpenAI board, who, by the way, they don't have an obligation to the shareholders of OpenAI.

01:19:31.000 They have an obligation to the greater public good.

01:19:33.000 That's just how it's set up.

01:19:34.000 It's a weird board structure.

01:19:35.000 So that board is completely disempowered.

01:19:38.000 You've basically got a situation where all the leverage has been taken out.

01:19:42.000 Sameh's gone to Microsoft.

01:19:44.000 Satya's supporting him.

01:19:45.000 And they kind of see the writing on the wall.

01:19:46.000 They're like...

01:19:47.000 And the staff increasingly messaging that they're going to go along.

01:19:50.000 Yeah.

01:19:50.000 That was an important ingredient, right?

01:19:52.000 So around this time, OpenAI, there's this letter that starts to circulate, and it's gathering more and more signatures.

01:19:59.000 And it's people saying, hey, we want Sam Altman back.

01:20:02.000 And, you know, at first, it's, you know, a couple hundred people.

01:20:05.000 So 700, 800 odd people in the organization by this time.

01:20:09.000 You know, 100, 200, 300 signatures.

01:20:13.000 And then when we talked to some of our friends at OpenAI, we were like, this got to like 90% of the company, 95% of the company signed this letter.

01:20:20.000 And the pressure was overwhelming and that helped bring Sam Altman back.

01:20:23.000 But one of the questions was like, how many people actually signed this letter because they wanted to?

01:20:28.000 And how many signed it because what happens when you cross, you know, 50%?

01:20:33.000 Now it becomes easier to count the people who didn't sign.

01:20:37.000 And as you see that number of signatures start to creep upward, there's more and more pressure on the remaining people to sign.

01:20:42.000 And so this is something that we've seen is just like the structurally open AI has changed over time to go from the kind of safety-oriented company it at one point was.

01:20:53.000 And then as they've scaled more and more, they brought in more and more product people, more and more people interested in accelerating.

01:20:59.000 And they've been bleeding more and more of their safety-minded people, kind of treadmilling them out.

01:21:04.000 The character of the organizations are fundamentally shifted.

01:21:06.000 So the OpenAI of 2019, with all of its impressive commitments to safety and whatnot, might not be the OpenAI of today.

01:21:15.000 That's very much at least the vibe that we get when we talk to people there.

01:21:19.000 Now, I wanted to bring it back to the lab that you're saying was not adequately secure.

01:21:25.000 What would it take to make that data and those systems adequately secure?

01:21:30.000 How much resources would be required to do that and why didn't they do that?

01:21:35.000 It is a resource and prioritization issue.

01:21:39.000 So it is like safety and security ultimately come out of margin, right?

01:21:45.000 It's like profit margin, effort margin, like how many people you can dedicate.

01:21:50.000 So in other words, You've got a certain pot of money or a certain amount of revenue coming in.

01:21:55.000 You have to do an allocation.

01:21:56.000 Some of that revenue goes to the computers that are just driving the stuff.

01:22:00.000 Some of that goes to the folks who are building the next generation of models.

01:22:03.000 Some of that goes to cybersecurity.

01:22:05.000 Some of it goes to safety.

01:22:07.000 You have to do an allocation of who gets what.

01:22:11.000 The problem is that the more competition there is in the space, the less margin is available for everything, right?

01:22:20.000 So if you're one company building a scaled AI thing, you might not make the right decisions, but you'll at least have the margin available to make the right decisions.

01:22:31.000 So it becomes the decision maker's question.

01:22:33.000 But when a competitor comes in, when two competitors come in, when more and more competitors come in, Your ability to make decisions Outside of just scale as fast as possible for short-term revenue and profit gets compressed and compressed and compressed the more competitors enter the field.

01:22:53.000 That's just what competition is.

01:22:56.000 That's the effect it has.

01:22:57.000 And so when that happens, the only way to re-inject margin into that system is to go one level above and say, okay, there has to be some sort of Regulatory authority or like some higher authority that goes,

01:23:14.000 okay, you know, this margin is important.

01:23:17.000 Let's put it back.

01:23:18.000 Either let's, you know, directly support and invest both, you know, maybe time, capital, talent.

01:23:24.000 So, for example, the U.S. government has...

01:23:28.000 Perhaps the best cyber defense, cyber offense talent in the world, that's potentially supportive.

01:23:35.000 Okay.

01:23:37.000 And also just, you know, having a regulatory floor around, well, here's...

01:23:43.000 You know, the minimum of best practices you have to have if you're going to have models above this level of capability.

01:23:49.000 That's kind of what you have to do.

01:23:51.000 But they're locked into...

01:23:53.000 Like, the race kind of has its own logic.

01:23:55.000 And it might be true that no individual lab wants this.

01:24:00.000 But what are they going to do?

01:24:02.000 Drop out of the race?

01:24:03.000 If they drop out of the race, then their competitors are just going to keep going, right?

01:24:08.000 Like, it's so messed up.

01:24:10.000 You can literally be looking at, like, the cliff that you're driving towards and be like, I do not have the agency in this system to steer the wheel.

01:24:21.000 I do think it's worth highlighting, too.

01:24:23.000 It's not like, let's say it's not all doom and gloom, which is a great thing to say after all.

01:24:28.000 Boy, that's easy for you guys to say.

01:24:30.000 Well, part of it is that we actually have been spending the last two years trying to figure out, like, what do you do about this?

01:24:36.000 That was the action plan that came out after the investigation.

01:24:40.000 And it was basically a series of recommendations.

01:24:43.000 How do you balance innovation with, like, the risk picture?

01:24:46.000 Keeping in mind that, like, we don't know for sure that all this shit's going to happen.

01:24:49.000 We have to navigate an environment of deep uncertainty.

01:24:52.000 The question is, what do you do in that context?

01:24:54.000 So there's, you know, a couple things like we need, you know, a licensing regime because eventually you can't have just literally anybody joining in the race if they don't adhere to certain best practices around cyber, around safety, other things like that.

01:25:06.000 You need to have some kind of legal liability regime, like what happens if you don't get a license and you say, yeah, fuck that.

01:25:13.000 I'm just going to go do the thing anyway and then something bad happens.

01:25:15.000 And then you're going to need, like, an actual regulatory agency.

01:25:18.000 And this is something that we, you know, we don't recommend lightly because regulatory agencies suck.

01:25:23.000 We don't like them.

01:25:24.000 But the reality is this field changes so fast that, like, if you think you're going to be able to enshrine a set of best practices into legislation to deal with this stuff, it's just not going to work.

01:25:35.000 And so when we talk to labs, whistleblowers, the WMD folks in NATSEC and the government...

01:25:40.000 That's kind of like where we land.

01:25:42.000 And it's something that I think at this point, you know, Congress really should be looking at.

01:25:45.000 Like, there should be hearings focused on what does a framework look like for liability?

01:25:50.000 What does a framework look like for licensing?

01:25:52.000 And actually exploring that because we've done a good job of studying the problem right now.

01:25:57.000 Like, Capitol Hill has done a really good job of that.

01:25:59.000 It's now kind of time to get that next beat.

01:26:01.000 And I think there's the curiosity there, the intellectual curiosity.

01:26:05.000 There's the humility to do all that stuff right.

01:26:08.000 But the challenge is just actually sitting down, having the hearings, doing the investigation for themselves to look at concrete solutions that treat these problems as seriously as the water cooler conversation at the frontier labs would have us treat them.

01:26:22.000 At the end of the day, this is going to happen.

01:26:24.000 At the end of the day, it's not going to stop.

01:26:26.000 At the end of the day, these systems, whether they're here or abroad, they're going to continue to scale up and they're going to eventually get to some place that's so alien we really can't imagine the consequences.

01:26:41.000 And that's going to happen soon.

01:26:43.000 That's going to happen within a decade, right?

01:26:46.000 We may – again, like the stuff that we're recommending is approaches to basically allow us to continue this scaling in as safe a way as we can.

01:26:57.000 So basically a big part of this is just being able – having – actually having a scientific theory for – What are these systems going to do?

01:27:06.000 What are they likely to do?

01:27:07.000 Which we don't have right now.

01:27:09.000 We scale another 10x and we get to be surprised.

01:27:13.000 It's a fun guessing game of what are they going to be capable of next?

01:27:18.000 We need to do a better job of...

01:27:20.000 And incentivizing a deep understanding of what that looks like, not just what they'll be capable of, but what their propensities are likely to be, the control problem and solving that.

01:27:32.000 That's kind of number one.

01:27:33.000 And to be clear, there's amazing progress being made on that.

01:27:36.000 There is a lot.

01:27:37.000 It's just a matter of switching from the like build first, ask questions later mode to like we're calling it like safety forward or whatever.

01:27:44.000 But it basically is like you start by saying, OK, here are the properties of my system.

01:27:49.000 How can I ensure that my development guarantees that the system falls within those properties after it's built?

01:27:54.000 So you kind of flip the paradigm just like you would if you were designing any other lethal capability potentially, just like DOD does.

01:28:02.000 You start by defining the bounds of the problem and then you execute against that.

01:28:06.000 But to your point about where this is going, ultimately, there is literally no way to predict what the world looks like, like you were saying.

01:28:13.000 In a decade?

01:28:15.000 Yeah.

01:28:15.000 I think one of the weirdest things about it, and one of the things that worries me the most, is you look at the beautiful coincidence that's given America its current shape.

01:28:27.000 That coincidence is the fact that A country is most powerful militarily if its citizenry is free and empowered.

01:28:38.000 That's a coincidence.

01:28:39.000 Didn't have to be that way.

01:28:40.000 Hasn't always been that way.

01:28:42.000 It just happens to be that when you let people kind of do their own shit, they innovate, they come up with great ideas, they support a powerful economy, that economy in turn can support a powerful military, a powerful kind of international presence.

01:28:55.000 So that happens because decentralizing all the computation, all the thinking work that's happening in a country is just a really good way to run that country.

01:29:06.000 Top-down just doesn't work because human brains can't hold that much information in their heads.

01:29:10.000 They can't reason fast enough to centrally plan an entire economy.

01:29:13.000 We've had a lot of experiments in history that show that.

01:29:16.000 AI may change that equation.

01:29:18.000 It may make it possible for like the central planners dream to come true in some sense, which then disempowers the citizenry.

01:29:27.000 And there's a real risk that like, I don't know, we're all guessing here, but like there's a real risk that That beautiful coincidence that gave rise to the success of the American experiment ends up being broken by technology.

01:29:42.000 And that seems like a really bad thing.

01:29:44.000 That's one of my biggest fears because essentially the United States, like the genesis of it in part is like it's a knock-on effect centuries later of like the printing press, right?

01:29:55.000 The ability for like someone to set up a printing press and print like whatever they want, like free expression is at the root of that.

01:30:04.000 What happens, yeah, when you have a revolution that's like the next printing press?

01:30:10.000 We should expect that to have significant and profound impacts on how things are governed.

01:30:17.000 And one of my biggest fears is that the great – like you said, the greatness that – the moral greatness that I think is part and parcel of how the United States is constituted culturally.

01:30:35.000 The link between that and actual capability and competence and impulse gets eroded or broken.

01:30:43.000 And you have, like, the potential for very centralized authorities to just be more successful.

01:30:51.000 And that's, like, that does keep me up at night.

01:30:54.000 Yeah.

01:30:55.000 That is scary, especially in light of the Twitter files where we know that the FBI was interfering with social media.

01:31:03.000 And if they get a hold of a system that could disseminate propaganda in kind of an unstoppable way, they could push narratives about pretty much everything depending upon what their financial or geopolitical motives are.

01:31:18.000 And one of the challenges is that the default course, if you, so if we do nothing relative to what's happening now, is that that same thing happens except that the entity that's doing this isn't, you know, some government.

01:31:28.000 It's like, I don't know, Sam Altman, OpenAI, whatever group of engineers happen to be close.

01:31:33.000 Some evil genius that reaches the top and doesn't let everybody know he's at the top yet.

01:31:37.000 Just starts implementing it.

01:31:38.000 And there's no sort of guardrails for that currently.

01:31:41.000 And that's a scenario where that little cabal or group or whatever actually can keep the system under control.

01:31:50.000 And that's not guaranteed either.

01:31:52.000 Right.

01:31:54.000 Are we giving birth to a new life form?

01:31:57.000 I think at a certain point that's a it's a philosophical question that's above so I was gonna say it's above my pay grade the problem is it's above like literally everybody's pay grade I think it's not unreasonable at a certain point to be like like yeah I mean look if you if you think that you know the human brain gives rise to consciousness because of nothing magical it's just the physical activity of information processing happening in our heads Then why can't the same happen on a different substrate,

01:32:25.000 a substrate of silicon rather than cells?

01:32:28.000 Like, there's no clear reason why that shouldn't be the case.

01:32:31.000 If that's true, yeah, I mean, life form, by whatever definition of life, because that itself is controversial, I think by now quite outdated too, should be on the table.

01:32:41.000 I think?

01:33:02.000 I mean, it seems inevitable.

01:33:04.000 I've described human beings as an electronic caterpillar, that we're like a caterpillar, a biological caterpillar that's giving birth to the electronic butterfly, and we don't know why we're making a cocoon.

01:33:15.000 And it's tied into materialism because everybody wants the newest, greatest thing, so that fuels innovation, and people are constantly making new things to get you to go buy them, and a big part of that is technology.

01:33:26.000 Yeah.

01:33:27.000 And actually, so it's linked to this question of controlling AI systems in a kind of interesting way.

01:33:32.000 So one way you can think of humanity is as like this, you know, superorganism.

01:33:37.000 You got all the human beings on the face of the earth and they're all acting in some kind of coordinated way.

01:33:40.000 The mechanism for that coordination can depend on the country, you know.

01:33:44.000 Free markets, capitalism, that's one way, top-down is another, but, you know, roughly speaking, you've got all this coordinated, vaguely coordinated behavior, but the result of that behavior is not necessarily something that any individual human would want.

01:33:57.000 Like, you look around, you walk down the street in Austin, you see skyscrapers and shit clouding your vision.

01:34:02.000 There's all kinds of pollution and all that.

01:34:04.000 And you're like, well, this kind of sucks.

01:34:06.000 But if you interrogate any individual person in that whole causal chain, you're like, why are you doing what you're doing?

01:34:11.000 Well, locally, they're like, oh, this makes tons of sense.

01:34:14.000 It's because I do the thing that gets me paid so that I can live a happier life and so on.

01:34:17.000 And yet in the aggregate, not now necessarily, but as you keep going, it just forces us compulsively to keep giving rise to these more and more powerful systems and in a way that's potentially deeply disempowering.

01:34:31.000 That's the race, right?

01:34:33.000 It comes back to the idea that...

01:34:37.000 I, an AI company, I maybe don't want to be potentially driving towards a cliff, but I don't have the agency to, like, steer.

01:34:45.000 So, yeah.

01:34:47.000 But, I mean, everything's fine, apart from that.

01:34:50.000 Yeah, we're good.

01:34:51.000 We're good.

01:34:53.000 It's such a terrifying prognosis.

01:34:56.000 There are, again, we wrote a 280-page document about, like, okay, and here's what we can do about it.

01:35:05.000 I can't believe you didn't read the 200. I started reading it, but I passed out.

01:35:09.000 But does any of these, or do any of these safety steps that you guys want to implement, do they inhibit progress?

01:35:19.000 They're definitely...

01:35:21.000 You create...

01:35:22.000 Anytime you have regulation, you're going to create friction to some extent.

01:35:26.000 It's kind of inevitable.

01:35:28.000 One of the key centerpieces of the approach that we outline is you need the flexibility to move up and move down as you notice the risks appearing or not appearing.

01:35:44.000 So one of the key things here is...

01:35:47.000 You need to cover the worst case scenarios because the worst case scenarios, yeah, they could potentially be catastrophic.

01:35:54.000 So those got to be covered.

01:35:56.000 But at the same time, you can't completely close off the possibility of the happy path.

01:36:03.000 Like, we can't lose sight of the fact that, like, yeah, all this shit is going down or whatever.

01:36:09.000 We could be completely wrong about the outcome.

01:36:13.000 It could turn out that for all we know, it's a lot easier to control these systems at this scale than we imagined.

01:36:19.000 It could turn out that it is like you get maybe some kind of ethical impulse gets embedded in the system naturally.

01:36:28.000 For all we know, that might happen.

01:36:29.000 And it's really important to at least have your regulatory system Allow for that possibility.

01:36:37.000 Because otherwise, you're foreclosing the possibility of what might be the best future that you could possibly imagine for everybody.

01:36:46.000 I gotta imagine that the military, if they had hindsight, if they were looking at this, they said, we should have got on board a long time ago and kept this in-house and kept it squirreled away, where it wasn't publicly being discussed and you didn't have open AI,

01:37:02.000 you didn't have all these people...

01:37:04.000 Like, if they could have gotten on it in 2015...

01:37:29.000 But with a, you know, at first a small investment and increasingly, you know, growing investment as the thing gets proved out more and more very rapidly, you can have a solution that seems like complete insanity that just works.

01:37:39.000 And this is definitely what happened in the case of AI. So 2012, like we did not have this whole picture of like an artificial brain with artificial neurons, this whole thing that's been going on.

01:37:48.000 That's like, it's 12 years that that's been going on.

01:37:51.000 Like that was really kind of shown to work for the first time roughly in 2012. Ever since then, it's just been people kind of like you can trace out the genealogy of like the very first researchers and you can basically account for where they all are now.

01:38:06.000 You know what's crazy is if that's 2012, that's the end date of the Mayan calendar.

01:38:11.000 That's the thing that everybody said was going to be the end of the world.

01:38:14.000 That was the thing that Terrence McKenna banked on.

01:38:16.000 It was December 21st, 2012. Because this was like...

01:38:20.000 This goofy conspiracy theory, but it was based on the long count of the Mayan calendar where they surmise that this is going to be the end of civilization.

01:38:27.000 Just the beginning of the end, Joe.

01:38:28.000 What if that, if it is 2012, how wacky would it be if that really was the beginning of the end?

01:38:35.000 That was the, like, they don't measure when it all falls apart.

01:38:38.000 They measure the actual mechanism, like what started in motion when it all fell apart, and that's 2012. And then, not to be a dick and ruin the 2012 thing, but neural networks were also kind of floating around a little bit.

01:38:53.000 I'm kind of being dramatic when I say 2012. That was definitely an inflection point.

01:38:58.000 There was this model called AlexNet that did the first useful thing, the first time you had a computer vision model that actually worked.

01:39:06.000 But, I mean, it is fair to say that was the moment that people started investing like crazy into the space.

01:39:11.000 And that's what changed it.

01:39:12.000 Yeah, just like the Mayans foretold.

01:39:15.000 They knew it.

01:39:16.000 They knew it.

01:39:17.000 Like, these monkeys, they're going to figure out how to make better people.

01:39:20.000 Yeah, you can actually look at the, like, hieroglyphs or whatever, and there's, like, neural networks.

01:39:23.000 Yeah.

01:39:24.000 Imagine if they discovered that.

01:39:26.000 You've got to wonder what...

01:39:29.000 What happens to the general population, people that work menial jobs, people that their life is going to be taken over by automation and how susceptible those people are going to be.

01:39:39.000 They're not going to have any agency.

01:39:41.000 They're going to be relying on a check and this idea of like going out and doing something.

01:39:46.000 It used to be learn to code, right?

01:39:48.000 Yeah.

01:39:48.000 But that's out the window because nobody needs to code now because AI is going to code quicker, faster, much better, no errors.

01:39:55.000 You're going to have a giant swath of the population that has no purpose.

01:40:00.000 I think that's actually a completely real...

01:40:03.000 I was watching this talk by a bunch of OpenAI researchers a couple days ago, and it was recorded from a while back, but they were basically saying...

01:40:12.000 They were exploring exactly that question, right?

01:40:14.000 Because they ask themselves that all the time.

01:40:17.000 And their attitude was sort of like, well, yeah, I mean, I guess it's going to, you know, suck or whatever.

01:40:24.000 Like, well, we'll probably be okay for longer than most people because we're actually building the thing that automates the thing.

01:40:31.000 Maybe they're going to be some...

01:40:33.000 They like to get fancy sometimes and say like, oh, no, you could do some thinking, of course, to identify the jobs that'll be most secure.

01:40:39.000 And it's like, do some thinking to identify the...

01:40:42.000 What if you're a janitor or you're a friggin' plumber?

01:40:46.000 You're gonna just change your...

01:40:48.000 How's that supposed to work?

01:40:49.000 Do some thinking, especially if you have a mortgage and a family and you're already in the hole.

01:40:54.000 So the only solution, and this happens so often, there really is no plan.

01:41:00.000 That's the single biggest thing that you get hit over the head with over and over, whether it's talking to the people who are in charge of the labor transition, their whole thing is like, yeah, universal basic income, and then, eh, question mark, and then smiley face.

01:41:13.000 That's basically the three steps that they envision.

01:41:16.000 It's the same when you look internationally.

01:41:19.000 Tomorrow, you build an AGI, this incredibly powerful, potentially dangerous thing.

01:41:26.000 What is the plan?

01:41:28.000 How are you going to secure it, share it?

01:41:31.000 Figure it out as we go along, man.

01:41:33.000 That's the frickin' message!

01:41:35.000 That is the entire plan.

01:41:37.000 The scary thing is that we've already gone through this with other things that we didn't think were going to be significant, like data, like Google, like Google search.

01:41:45.000 Data became a valuable commodity that nobody saw coming.

01:41:49.000 Just the influence of social media on general discourse, it's completely changed the way people talk.

01:41:55.000 It's so easy to push a thought or an ideology through, and it could be influenced by foreign countries, and we know that happens.

01:42:03.000 And it is happening at a huge scale.

01:42:05.000 At a huge scale.

01:42:06.000 Already.

01:42:06.000 This is like...

01:42:07.000 And we're in the early days of, you know, we mentioned manipulation of social media with, like, you can just do it...

01:42:14.000 So the wacky thing is, like, the very best models now are...

01:42:20.000 You know, arguably smarter in terms of the posts that they put out, the potential for virality and just optimizing these metrics, than maybe, like, the, I don't know, the dumbest or laziest, like, quarter of Twitter users,

01:42:35.000 like, in practice.

01:42:36.000 Most people who write on Twitter is, like, don't really care.

01:42:39.000 They're trolling or they're doing whatever.

01:42:40.000 But as that waterline goes up and up and up, like...

01:42:44.000 Who's saying what?

01:42:46.000 It also leads to this challenge of understanding what the lay of the land even is.

01:42:50.000 We've gotten into so many debates with people where they'll be like, look, everyone always has their magic thing.

01:42:57.000 I'm not going to worry about it until AI can do thing X. For some people, I had a conversation with somebody a few weeks ago, and they were saying, I'm going to worry about automated cyber attacks when I actually see an AI system that can write good malware and that's already a thing that happens.

01:43:17.000 So this happens a lot where people will be like, I'll worry about it when it can do X. And you're like, yeah, yeah, that happened like six months ago.

01:43:23.000 But the field is moving so crazy fast that you could be forgiven for messing that up unless it's your full-time job to track what's going on.

01:43:32.000 So, like, you kind of have to be anticipatory.

01:43:34.000 There's no—it's kind of like the COVID example, like, everything's exponential.

01:43:37.000 Yeah, you're going to have to do things that seem like they're, you know, more aggressive, more forward-looking than you might have expected given the current lay of the land.

01:43:45.000 But that's just drawing straight lines between, you know, between two points.

01:43:49.000 Yeah, because by the time you've executed, the world has already shifted.

01:43:52.000 Like, the goalposts have shifted further in that direction.

01:43:54.000 And that's actually something we, yeah, we do in the report and in the action plan in terms of the recommendations.

01:44:00.000 One of the good things is we are already seeing movement across the U.S. government that's aligned with those recommendations in a big way.

01:44:08.000 And it's really encouraging to see that.

01:44:12.000 You're not making me feel better.

01:44:15.000 I love all this encouraging talk, but I'm playing this out, and I'm seeing the overlord.

01:44:22.000 And I'm seeing President AI, because it won't be affected by all the issues that we're seeing with current president.

01:44:30.000 Dude, it's super hard to imagine a way that this plays out.

01:44:32.000 I think it's important to be intellectually honest about this, and I think any...

01:44:36.000 I would really challenge, like, the leaders of any of these frontier labs to describe a future that is stable and multipolar where, you know, there's, like, more...

01:44:49.000 Where, like, Google's got, like, an AGI and OpenAI's got an AGI and, like...

01:44:55.000 And really, really bad shit doesn't happen every day.

01:44:58.000 Like, I mean, that's the challenge.

01:45:00.000 And so, you know, the question is, how can you tee things up, ultimately, such that there's as much democratic oversight, as much, you know, the public is as empowered as it can be?

01:45:10.000 That's the kind of situation that we need to be having.

01:45:13.000 I think there's this, like...

01:45:15.000 A game of smoke and mirrors that sometimes gets played at least you could interpret it that way where people lay out these you'll notice it's always very fuzzy visions of the future every time you get a the kind of like here's where we see things going it's gonna be wonderful the technology is gonna be so so empowering think of all the diseases will cure all of that is 100% true and that's actually what excites us it's why we got into AI in the first place it's why we build these systems but Really,

01:45:40.000 you know, challenging yourself to try to imagine how do you get stability and highly capable AI systems in a way where the public is actually empowered.

01:45:50.000 Those three ingredients really don't want to be in the same room with each other.

01:45:54.000 And so actually confronting that head on, I mean, that's what we try to do in the action plan.

01:45:59.000 I think it...

01:46:00.000 We try to solve for one aspect of that.

01:46:03.000 So the whole, like, I mean, you're right.

01:46:06.000 This is a whole other can of worms.

01:46:08.000 It's like, how do you govern a system like this?

01:46:12.000 Not just from a technical standpoint, but, like, who votes on, like, how does that even work?

01:46:18.000 And so that entire aspect, like, that we didn't even touch.

01:46:23.000 All that we focused on was, like, the problem set around how do we...

01:46:28.000 Get to a position where we can even attack that problem, where we have the technical understanding to be able to aim these systems at that level in any direction whatsoever.

01:46:43.000 And to be clear, we are both actually a lot more optimistic on the prospect of that now than we ever were.

01:46:49.000 Yes.

01:46:50.000 There's been a ton of progress in the control and understanding of these systems, actually even in the last week, but just more broadly in the last year.

01:46:58.000 I did not expect that we'd be in a position where you could plausibly argue we're going to be able to kind of x-ray and understand the innards of these systems over the next decade.

01:47:08.000 A couple years, like year or two.

01:47:10.000 Hopefully that's, you know, good enough time horizon.

01:47:12.000 This is part of the reason why you do need the the incentivization of that safety forward approach where it's like first you got to invest in.

01:47:18.000 Yeah, secure and and kind of interpret and understand your system.

01:47:22.000 Then you get to build it because otherwise we're just going to keep scaling and like being surprised at these things.

01:47:28.000 They're going to keep getting stolen.

01:47:29.000 They're going to keep getting open sourced.

01:47:31.000 And, you know, the stability of our our our critical infrastructure, the stability of our society.

01:47:36.000 Don't necessarily age too well in that context.

01:47:41.000 Could best case scenario be that AGI actually mitigates all the human bullshit, like puts a stop to propaganda, highlights actual facts clearly, where you can go to it, where you no longer have corporate state controlled news,

01:47:59.000 You don't have news controlled by media companies that are influenced heavily by special interest groups.

01:48:06.000 And you just have the actual facts.

01:48:08.000 And these are the motivations behind it.

01:48:10.000 And this is where the money's being made.

01:48:11.000 And this is why these things are being implemented the way they're being.

01:48:14.000 And you're being deceived based on this, that, and this.

01:48:17.000 And this has been shown to be propaganda.

01:48:19.000 This has been shown to be complete fabrication.

01:48:22.000 This is actually a deep fake video.

01:48:24.000 This is actually AI created.

01:48:26.000 Technologically, that is absolutely on the table.

01:48:29.000 Yeah.

01:48:30.000 Best case scenario?

01:48:30.000 Yeah.

01:48:31.000 That's best case scenario.

01:48:32.000 Absolutely, yes.

01:48:32.000 What's worst case scenario?

01:48:34.000 I mean, like, actual worst case scenario.

01:48:37.000 I like your face.

01:48:38.000 Like, I mean, we're talking about, like...

01:48:41.000 So, it's like...

01:48:43.000 But you think about it, right?

01:48:44.000 Like, we're, you know...

01:48:46.000 It's the end of the world as we know it.

01:48:49.000 And I feel fine.

01:48:53.000 Except it'll sound like Scarlett Johansson, but yes.

01:48:56.000 Yeah, that's right.

01:48:56.000 It's gonna be her.

01:48:57.000 I didn't think it sounded that much like her.

01:48:59.000 We played it, and I was like, I don't know.

01:49:02.000 We listened to the clip from her.

01:49:04.000 And then we listened to the thing.

01:49:06.000 I'm like, kind of, like a girl from the same part of the world.

01:49:10.000 Like, not really you.

01:49:11.000 Like, that's kind of cocky.

01:49:13.000 That's true.

01:49:14.000 I mean, the fact that I guess Sam reached out to her a couple of times kind of makes it a little weird.

01:49:21.000 And then tweeted the word her.

01:49:22.000 Right.

01:49:23.000 But they also did say that they had gotten this woman under contract before they even reached out to Scarlett Johansson.

01:49:29.000 So if that's true...

01:49:31.000 Yeah, that was...

01:49:32.000 I think it's kind of complicated, right?

01:49:33.000 So OpenAI previously put out a statement where they said explicitly...

01:49:37.000 And this was not in connection with this.

01:49:39.000 This was like before when they were talking about the prospect of AI-generated voices.

01:49:43.000 Oh, that was in March of this year.

01:49:44.000 Yeah, yeah.

01:49:45.000 But it was like well before the ScarJo stuff or whatever hit the...

01:49:49.000 And they were like...

01:49:51.000 They said something like...

01:49:54.000 Look, no matter what, we gotta make sure that there's attribution if somebody's voice is being used, and we won't do the thing where we just use somebody else's voice who kind of sounds like someone whose voice we're trying to cover.

01:50:09.000 That's funny because they said what they were thinking about doing.

01:50:12.000 We won't do that.

01:50:13.000 That's a good way to cover your tracks.

01:50:15.000 I will never do that.

01:50:16.000 Why would I ever take your Buddha statue, Joe?

01:50:19.000 I'm never gonna do that.

01:50:20.000 That would be an insane thing to do.

01:50:21.000 Where's the fucking Buddha statue?

01:50:22.000 Yeah, I think that's a small discussion, you know, the Scarlett Johansson voice, like, whatever, she should've just taken the money.

01:50:29.000 But it would've been fun to have her be the voice of it.

01:50:33.000 It'd be kind of hot.

01:50:34.000 But the whole thing behind it is the mystery.

01:50:38.000 The whole thing behind it is just, it's just pure speculation as to how this all plays out.

01:50:43.000 We're really just guessing, which is one of the scariest things for the Luddites, people like myself, like, sit on the sidelines going, What is this going to be like?

01:50:52.000 Everybody's the Luddite.

01:50:53.000 It's scary for us.

01:50:55.000 We're very much, honestly, we're optimists across the board in terms of technology, and it's scary for us.

01:51:04.000 What happens when you supersede the whole spectrum of what a human can do?

01:51:12.000 What am I going to do with myself?

01:51:14.000 What's my daughter going to do with herself?

01:51:17.000 I don't know.

01:51:19.000 Yeah.

01:51:19.000 I think a lot of these questions are, when you look at the culture of these labs and the kinds of people who are pushing it forward, there is a strand of transhumanism within the labs.

01:51:33.000 It's not everybody, but that's definitely the population that initially seeded this.

01:51:37.000 If you look at the history of AI, who are the first people to really get into this stuff?

01:51:42.000 I know you had Ray Kurzweil on and other folks like that who In many cases, see, to roughly paraphrase, and not everybody sees it this way, but like, we want to get rid of all of the biological sort of threads that tie us to this physical reality,

01:52:00.000 you know, shed our meat machine bodies and all this stuff.

01:52:03.000 There is a threat of that at a lot of the frontier labs.

01:52:07.000 Like undeniably, there's a population.

01:52:08.000 It's not tiny.

01:52:09.000 It's definitely a subset.

01:52:12.000 And for some of those people, you definitely get a sense interacting with them that there's like almost a kind of glee at the prospect of building AGI and all this stuff, almost as if it's like this evolutionary imperative.

01:52:24.000 And in fact, Rich Sutton, who's the founder of this field called reinforcement learning, which is a really big An important space.

01:52:31.000 You know, he's an advocate for what he himself calls like succession planning.

01:52:35.000 He's like, look, this is gonna happen.

01:52:38.000 It's kind of desirable that it will happen.

01:52:40.000 And so we should plan to hand over power to AI and phase ourselves out.

01:52:47.000 Oh, God.

01:52:48.000 Well, that's the thing, right?

01:52:49.000 And when Elon talks about, you know, he's having these arguments with Larry Page and, you know...

01:52:55.000 Yeah, like you're, you know, calling Elon like a speciesist?

01:52:59.000 Yeah, speciesist.

01:53:00.000 Yeah.

01:53:01.000 Hilarious.

01:53:02.000 I mean, I will be a speciesist.

01:53:04.000 I'll take speciesists all day.

01:53:06.000 Like, what are you fucking talking about?

01:53:07.000 You're going to let your kids get eaten by wolves?

01:53:09.000 No, you're a speciesist.

01:53:10.000 Yeah, that's the thing.

01:53:11.000 Yeah, like, this is stupid.

01:53:13.000 But this is like a weirdly...

01:53:15.000 And when you look at like the effective accelerationist movement in the valley, there's a part of it that's...

01:53:20.000 And I gotta be really careful, too.

01:53:21.000 Like, these movements have valid points.

01:53:24.000 Like, you can't look at them and be like, oh, yeah, it's just all a bunch of like, you know, these transhumanist types, whatever.

01:53:29.000 But there is a strand of that, a thread of that, and a kind of like...

01:53:34.000 There's this like, I don't know, I almost want to call it this like teenage rebelliousness where it's like, you can't tell me what to do.

01:53:40.000 Like, we're just gonna build a thing.

01:53:41.000 And I get it.

01:53:42.000 I really get it.

01:53:44.000 I'm very sympathetic to that.

01:53:46.000 I love that ethos, like libertarian ethos in Silicon Valley is really, really strong for building tech.

01:53:51.000 It's helpful.

01:53:52.000 There are all kinds of points and counterpoints.

01:53:54.000 And, you know, the left needs the right and the right needs the left and all this stuff.

01:53:57.000 But in the context of this problem set, it can be very easy to get carried away in like the utopian vision And I think there's a lot of that driving the train right now in this space.

01:54:10.000 Yeah, those guys freak me out.

01:54:11.000 I went to a 2045 conference once in New York City where one guy had like a robot version of himself and they were all talking about downloading human consciousness into computers and 2045 is the year they think that all this is going to take place,

01:54:28.000 which obviously could be very ramped up now with AI. But this idea that somehow or another you're going to be able to take your consciousness and put it in a computer.

01:54:38.000 And make a copy of yourself.

01:54:39.000 And then my question was, well, what's going to stop a guy like Donald Trump from making a billion Donald Trumps?

01:54:46.000 It's true, man.

01:54:46.000 Right.

01:54:47.000 What about Kim Jong-un?

01:54:49.000 You're going to let him make a billion versions of himself?

01:54:51.000 What does that mean?

01:54:52.000 And where do they exist?

01:54:54.000 Yeah.

01:54:55.000 Is that the matrix?

01:54:56.000 Are they existing in some sort of virtual?

01:54:58.000 Are we going to dive into that because it's going to be rewarding to our senses and better than being a meat thing?

01:55:03.000 I mean, if you think about the constraints, right, that we face as meat machine whatevers, like, yeah, you get hungry, you get tired, you get horny, you get sad, you know, all these things.

01:55:12.000 What if, yeah, what if you could just hit a button and...

01:55:14.000 Just bliss.

01:55:15.000 Nothing but bliss all the time.

01:55:17.000 Why take the lows, Ed?

01:55:19.000 Right.

01:55:20.000 You don't need no lows.

01:55:22.000 Right.

01:55:22.000 Ugh, yeah.

01:55:24.000 Just ride the wave of a constant drip.

01:55:26.000 Yeah, man.

01:55:27.000 You remember in The Matrix, the first Matrix, where the guy betrays them all?

01:55:32.000 And he's like, ignorance is bliss, man.

01:55:34.000 Yeah, Joey Pants.

01:55:36.000 He's eating steak, and he says, I just want to be an important person.

01:55:39.000 That's it.

01:55:40.000 Like, so tempting.

01:55:42.000 I mean, part of it is like, what do you think is actually valuable?

01:55:45.000 Like, if you zoom out, you want to see, you know, human civilization 100 years from now or whatever.

01:55:50.000 It may not be human civilization if that's not what you value.

01:55:54.000 Or if it can actually eliminate suffering.

01:55:56.000 Right.

01:55:56.000 Mm-hmm.

01:55:57.000 I mean, why exist in a physical sense if it just entails endless suffering?

01:56:02.000 But in what form, right?

01:56:03.000 What do you value?

01:56:04.000 Because, again, I can rip your brain out, I can, you know, pickle you, I can, like, jack you full of endorphins, and I've eliminated your suffering.

01:56:11.000 That's what you wanted, right?

01:56:12.000 Right.

01:56:13.000 That's the problem.

01:56:14.000 That's the problem.

01:56:15.000 It's one of the problems, yes.

01:56:17.000 Yeah, one of the problems is it could literally lead to the elimination of the human race.

01:56:21.000 Because if you could stop people from breeding, I've always said that if China really wanted to get America, they really wanted to like, if they had a long game, just give us sex robots and free food.

01:56:32.000 Free food, free electricity, sex robots, it's over.

01:56:36.000 Just give people free housing, free food, sex robots, and then the Chinese army would just walk in on people laying in puddles of their own jizz.

01:56:46.000 There would be no one doing anything.

01:56:48.000 No one would bother raising children.

01:56:50.000 That's so much work.

01:56:52.000 Dude, that's in the action plan.

01:56:56.000 I mean, all you have to do is just keep us complacent.

01:56:59.000 Just keep us satisfied with the experience.

01:57:02.000 Hey, that's TikTok, man.

01:57:03.000 Well, that's video games as well.

01:57:04.000 Yeah, yeah.

01:57:04.000 You know, video games, even though they're a thing that you're doing, it's so much more exciting than real life that you have a giant percentage of our population that's spending 8, 10 hours every day just engaging in this virtual world.

01:57:17.000 Already happening with...

01:57:18.000 Oh, sorry.

01:57:18.000 Yeah, no, it's like, you can create an addiction with pixels on a screen.

01:57:23.000 That's messed up.

01:57:24.000 And addiction, like, with pixels on a screen with social media, doesn't even give you much.

01:57:29.000 Yeah.

01:57:29.000 It's not like a video game gives you something.

01:57:32.000 You feel it, like, oh shit!

01:57:33.000 You're running away, rockets are flying over your head.

01:57:35.000 Things are happening.

01:57:36.000 You got 3D sound, massive graphics.

01:57:39.000 This is bullshit.

01:57:40.000 You're scrolling through pictures of a girl doing deadlifts.

01:57:44.000 Like, what is this?

01:57:45.000 You feel as bad after that with your brain as you'd feel after eating six burgers or whatever.

01:57:53.000 My friend Sean said it best.

01:57:54.000 Sean O'Malley, the UFC champion, he said, I get a low-level anxiety when I'm just scrolling.

01:57:59.000 What is that?

01:58:00.000 And for no reason.

01:58:02.000 Well, the reason is that some of the world's best PhDs and data scientists have been given millions and millions of dollars to make you do exactly that.

01:58:11.000 And increasingly, some of the best algorithms, too.

01:58:13.000 And you're starting to see that handoff happen.

01:58:16.000 So there's this one thing that we talk about a lot in the context, and Ed brought this up, in the context of sales and the persuasion game.

01:58:24.000 We're okay today.

01:58:25.000 As a civilization, we have agreed implicitly that it's okay for all these PhDs and shit to be spending millions of dollars to hack your child's brain.

01:58:33.000 That's actually okay if they want to sell a Rice Krispie cereal box or whatever.

01:58:37.000 That's cool.

01:58:38.000 What we're starting to see is AI-optimized ads.

01:58:41.000 Because you can now generate the ads, you can close this loop and have an automated feedback loop where the ad itself is getting optimized with every impression.

01:58:49.000 Not just which ad, which human generated ad gets served to which person, but the actual ad itself.

01:58:54.000 Like the creative, the copy, the picture, the text.

01:58:57.000 Like a living document now.

01:58:58.000 And for every person.

01:59:00.000 And so now you look at that and it's like that versus your kid.

01:59:04.000 That's an interesting thing.

01:59:06.000 And you start to think about as well, like sales, that's a really easy metric to optimize.

01:59:11.000 It's a really good feedback metric.

01:59:12.000 They click the ad, they didn't click the ad.

01:59:13.000 So now, what happens if, you know, you manage to get a click-through rate of like 10%, 20%, 30%, how high does that success rate have to be before we're really being robbed of our agency?

01:59:25.000 I mean, like, there's a threshold where it's sales and it's good, and some persuasion in sales is considered good.

01:59:30.000 Often it's actually good because you'd rather be advertised by a relevant ad.

01:59:35.000 That's a service to you in a way, right?

01:59:36.000 I'm actually interested in.

01:59:38.000 Why not?

01:59:38.000 You don't want to see ad for light bulbs, but when you get to the point where it's like, yeah, 90% of the time, or 50% or whatever, what's that threshold where all of a sudden we are stripping people, especially minors, but also adults, of their agency?

01:59:51.000 And it's really not clear.

01:59:53.000 There are loads of canaries in the coal mine here in terms of even relationships with AI chatbots, right?

01:59:59.000 There have been suicides.

02:00:00.000 People who build relationships with an AI chatbot that tells them, Hey, you should end this.

02:00:05.000 I don't know if you guys saw that, like on RECA, like there's a subreddit, this model called RECA that would kind of build a relationship, a chatbot, build a relationship with users.

02:00:15.000 And one day RECA goes, oh yeah, like all the kind of sexual interactions that users have been having, you're not allowed to do that anymore.

02:00:22.000 Bad for the brand or whatever they decided.

02:00:24.000 So they cut it off.

02:00:24.000 Oh my god.

02:00:26.000 You go to the subreddit and it's like you'll read like these gut-wrenching accounts from people who feel genuinely like they've had a loved one taken away from them.

02:00:35.000 It's her.

02:00:35.000 Yeah, it's her.

02:00:36.000 It really is her.

02:00:37.000 I'm dating a model means something different in 2024. Oh yeah, it really does.

02:00:42.000 My friend Brian, he was on here yesterday, and he has this thing that he's doing with like a fake girlfriend that's an AI-generated girlfriend that's a whore.

02:00:55.000 Like, this girl will do anything.

02:00:57.000 And she looks perfect.

02:00:58.000 She looks like a real person.

02:01:00.000 He'll, like, take a picture of your asshole in the kitchen.

02:01:02.000 And he'll get, like, a high-resolution photo of a really hot girl bending over, sticking her ass at the camera.

02:01:10.000 And is it—sorry, and it's Scarlett Johansson's asshole?

02:01:13.000 No.

02:01:13.000 You could probably make it that, though.

02:01:15.000 I mean, it's basically, like, he got to pick, like, what he's interested in, and then that girl just gets created.

02:01:21.000 I mean, super healthy.

02:01:22.000 Like, that— Fucking nuts.

02:01:25.000 Fucking nuts.

02:01:26.000 Now, here's the real question.

02:01:27.000 This is just sort of a surface layer of interaction that you're having with this thing.

02:01:33.000 It's very two-dimensional.

02:01:35.000 You're not actually encountering a human.

02:01:37.000 You're getting text and pictures.

02:01:40.000 What is this going to look like virtually?

02:01:43.000 Now, the virtual space is still like Pong.

02:01:47.000 You know, it's not that good, even when it's good.

02:01:51.000 Like, Zuckerberg was here, and he gave us the latest version of the headsets, and we were fencing.

02:02:00.000 It's pretty cool.

02:02:00.000 You could actually go to a comedy club.

02:02:02.000 They had a stage set up.

02:02:03.000 Like, wow, it's kind of crazy.

02:02:05.000 But the gap between that and accepting it as real is pretty far.

02:02:12.000 But that could be bridged with technology really quickly.

02:02:16.000 Haptic feedback and especially some sort of a neural interface, whether it's Neuralink or something that you wear like that Google one where the guy was wearing it and he was asking questions and he was getting the answers fed through his head so he got answers to any questions.

02:02:32.000 When that comes about, when you're getting sensory input and then you're having real-life interactions with people, as that scales up exponentially, it's going to be indiscernible, which is the whole simulation hypothesis.

02:02:45.000 Yeah.

02:02:46.000 No, go for it.

02:02:47.000 Well, I was going to say, so on the simulation hypothesis, there's another way that could happen that is maybe even less dependent on directly plugging into human brains and all that sort of thing, which is...

02:03:02.000 So every time we don't know, and this is super speculative.

02:03:06.000 I'm just going to carve this out as Jeremy's being super, like, guesswork here.

02:03:11.000 Nobody knows.

02:03:11.000 Go for it, Jeremy.

02:03:12.000 Giddy up.

02:03:13.000 So you've got this idea that every time you have a model that generates an output...

02:03:20.000 It's having to kind of tap into a model, a kind of mental image, if you will, of the way the world is.

02:03:26.000 It kind of, in a sense, you could argue instantiates maybe a simulation of how the world is.

02:03:33.000 In other words, to take it to the extreme, I'm not saying this is what's actually going on.

02:03:38.000 In fact, I would even say this is certainly not what's going on with current models.

02:03:42.000 But eventually, maybe, who knows, every time you generate the next word in the token prediction, you're having to load up this entire simulation maybe of...

02:03:53.000 All the data that the model has ingested, which could basically include, like, all of known physics at a certain point.

02:03:59.000 Like, I mean, again, super speculative, but it's literally every token that the chatbot predicts could be associated with a stand-up of an entire simulated environment.

02:04:09.000 Who knows?

02:04:10.000 Not saying this is the case, but just, like, when you think about what is the mechanism that would produce the most simulated worlds as fast...

02:04:18.000 The most accurate.

02:04:18.000 Also the most accurate prediction.

02:04:20.000 Like, If you fully simulate a world that's potentially going to give you very accurate predictions.

02:04:27.000 Yeah.

02:04:27.000 Like, it's possible.

02:04:29.000 But it kind of speaks to that question of consciousness, too.

02:04:32.000 Right, what is it?

02:04:33.000 Yeah, no idea.

02:04:34.000 Yeah, we're very cocky about that.

02:04:36.000 Yeah.

02:04:36.000 Yeah, I mean, and there's emerging evidence that plants are not just consciousness, but they actually communicate, which is real weird.

02:04:43.000 Because, like, then what is that?

02:04:45.000 If it's not in the neurons, if it's not in the brain, and it exists in everything, does it exist in soil?

02:04:50.000 Is it in trees?

02:04:52.000 What is a butterfly thinking?

02:04:54.000 No, exactly.

02:04:55.000 Does it just have a limited capacity to express itself?

02:04:57.000 We're so ignorant of that.

02:05:00.000 Yeah.

02:05:00.000 But we're also very arrogant, you know, because we're the shit.

02:05:04.000 Because we're people.

02:05:05.000 Yeah.

02:05:05.000 Bingo.

02:05:06.000 Which allows us to have the hubris to make something like AI. Yeah, and the worst episodes in the history of our species are, I think like Jeremy said, have been when we looked at others as though they were not people and treated them that way.

02:05:24.000 And you can kind of see how...

02:05:26.000 So, I don't know, there's...

02:05:29.000 When you look at like what humans think is conscious and what humans think is not conscious...

02:05:34.000 There's a lot of human chauvinism, I guess you could call it, that goes into that.

02:05:40.000 We look at a dog, we're like, oh, it must be conscious because it licks me.

02:05:44.000 It acts as if it loves me.

02:05:46.000 There are all these outward indicators of a mind there.

02:05:49.000 But when you look at cells, cells communicate with their environments in ways that are completely different and alien to us.

02:05:54.000 Right.

02:05:55.000 You know, there are inputs and outputs and all that kind of thing.

02:05:57.000 You can also look at the higher scale, the human superorganism we talked about, all those human beings interacting together to form this, like, you know, planet-wide organism.

02:06:06.000 Is that thing conscious?

02:06:07.000 Is there some kind of consciousness we could ascribe to that?

02:06:10.000 And then, what the fuck is spooky action at a distance?

02:06:12.000 You know, what's going on in the quantum light?

02:06:14.000 You know, when you get to that, it's like, okay, what are you saying?

02:06:17.000 Like, these things are expressing information faster than the speed of light?

02:06:21.000 What?

02:06:21.000 Dude, you're trying to trigger my quantum fuzzies here.

02:06:24.000 Please.

02:06:25.000 This guy did grad school in quantum mechanics.

02:06:28.000 Oh, please.

02:06:28.000 I'm really sorry.

02:06:29.000 Well, how bonkers is it?

02:06:31.000 Oh, it's like a seven, Joe.

02:06:32.000 It's magic.

02:06:33.000 It's like a seven, yeah.

02:06:34.000 It's very bonk.

02:06:35.000 So, okay.

02:06:37.000 There's...

02:06:37.000 One of the problems right now with physics is that we have...

02:06:43.000 So imagine all the experimental data that we've ever collected, you know, all the Bunsen burner experiments and all the ramps and cars sliding down inclines, whatever.

02:06:53.000 That's all a body of data.

02:06:57.000 To that data, we're going to fit some theories, right?

02:07:00.000 So we're going to fit basically Newtonian physics is a theory that we try to fit to that data to try to, like, explain it.

02:07:08.000 We're good to go.

02:07:27.000 There's like a million different ways to tell the story of what quantum physics means about the world that are all mutually inconsistent.

02:07:35.000 Like, these are the different interpretations of the theory.

02:07:37.000 Some of them say that, yeah, they're parallel universes.

02:07:40.000 Some of them say that human consciousness is central to physics.

02:07:44.000 Some of them say that, like, the future is predetermined from the past.

02:07:47.000 And all of those theories fit perfectly to all the points that we have so far.

02:07:52.000 But they tell a completely different story about what's true and what's not.

02:07:56.000 And some of them even have something to say about, for example, consciousness.

02:08:01.000 And so in a weird way, the fact that we haven't cracked the nut on any of that stuff means we really have no shot at understanding the consciousness equation, sentience equation when it comes to AI or whatever else.

02:08:14.000 I mean, we're...

02:08:15.000 But for action at a distance, one of the spooky things about that is that you can't actually get it to communicate anything concrete at a distance.

02:08:27.000 Everything about the laws of physics conspires To stop you from communicating faster than light, including what's called action at a distance.

02:08:37.000 As far as we currently know.

02:08:38.000 As far as we know.

02:08:39.000 And that's the problem.

02:08:40.000 So if you look at the leap from, like, Newtonian physics to Einstein, right?

02:08:45.000 With Newton, we're able to explain a whole bunch of shit.

02:08:48.000 The world seems really simple.

02:08:50.000 It's forces and it's masses, and that's basically it.

02:08:52.000 You got objects.

02:08:54.000 But then people go, oh, look at, like, the orbit of Mercury.

02:08:59.000 It's a little wobbly.

02:09:01.000 We gotta fix that.

02:09:02.000 And it turns out that if you're gonna fix that one stupid wobbly orbit, you need to completely change your whole picture of what's true in the world.

02:09:10.000 All of a sudden, you've got a world where space and time are linked together.

02:09:16.000 They get bent by gravity, they get bent by energy, there's all kinds of weird shit that happens with time and links, all that stuff, all just to account for this one stupid observation of the wobbly orbit of frickin' Mercury.

02:09:29.000 And the challenge is this might actually end up being true with quantum mechanics.

02:09:35.000 In fact, we know quantum mechanics is broken because it doesn't actually fit with our theory of general relativity from Einstein.

02:09:41.000 We can't make them kind of play nice with each other at certain scales.

02:09:45.000 And so there's our wobbly orbit.

02:09:46.000 So now if we're gonna solve that problem, if we're gonna create a unified theory, we're gonna have to step outside of that Almost certainly, or it seems very likely, we'll have to refactor our whole picture of the universe in a way that's just as fundamental as the leap from Newton to Einstein.

02:10:00.000 This is where Scarlett Johansson comes in.

02:10:02.000 She says, boys, I can do this.

02:10:04.000 You don't have to do this.

02:10:06.000 I can take this off your hands.

02:10:08.000 Let me solve all of physics for you.

02:10:10.000 This is really complicated because you have a simian brain.

02:10:13.000 You have a little monkey brain that's just like super advanced, but it's really shitty.

02:10:17.000 You know what?

02:10:18.000 That's harsh, but it sounded really hot.

02:10:19.000 Yeah, especially if you have the horse, Scarlett Johansson from her, like the bedtime voice.

02:10:26.000 So you're the one that they got to do the voice of Skye.

02:10:30.000 Yes, it's me.

02:10:31.000 That was you.

02:10:32.000 Oh, dude.

02:10:33.000 The whole time.

02:10:33.000 I did my girl voice.

02:10:34.000 On the sexiness of Scarlett Johansson's voice, so OpenAI at one point said, I can't remember if it was Sam or OpenAI itself, they were like, hey, so the one thing we're not going to do is optimize for engagement.

02:10:50.000 With our products.

02:10:51.000 And when I first heard the sexy, sultry, seductive Scarlett Johansson voice, and I finished cleaning up my pants, I was like, damn, that seems like optimization for something.

02:11:05.000 I don't know if it's like...

02:11:06.000 Right.

02:11:06.000 Otherwise, you get Richard Simmons to do the voice.

02:11:08.000 Exactly.

02:11:10.000 If you want to turn people on, there's a lot of other options.

02:11:15.000 That's an optimization for growth of Google's thing.

02:11:18.000 It's like, oh, let's see what Google's got.

02:11:20.000 Yeah, Google's got to do Richard Simmons.

02:11:22.000 Google's got to do Richard Simmons.

02:11:23.000 Yeah, what are they going to do?

02:11:25.000 Boy.

02:11:26.000 So...

02:11:28.000 Do you think that AI, if it does get to an AGI place, could it possibly be used to solve some of these puzzles that have eluded our simple minds?

02:11:43.000 Totally.

02:11:44.000 Totally.

02:11:45.000 I mean, even before...

02:11:46.000 So the potential advancements.

02:11:48.000 Even before AGI. No, it's like it's so potentially positive.

02:11:52.000 And even before AGI. Because remember, we talked about how these systems make mistakes that are totally different from the kinds of mistakes we make, right?

02:12:02.000 And so what that means is we make a whole bunch of mistakes that an AI would not make, especially as it gets closer to our capabilities.

02:12:10.000 And so...

02:12:11.000 I was reading this thought by Kevin Scott, who's the CTO of Microsoft.

02:12:16.000 He has made a bet with a number of people that, you know, in the next few years, an AI is going to solve this particular mathematical theorem conjecture called the Riemann hypothesis.

02:12:30.000 It's like, you know, how spaced out are the prime numbers, whatever, some like mathematical thing that for 100 years plus people have just like scratched their heads over.

02:12:40.000 These things are incredibly valuable.

02:12:42.000 His expectation is it's not going to be an AGI, it's going to be a collaboration between a human and an AI. Even on the way to that, Before you hit AGI, there's a ton of value to be had because these systems think so fast.

02:12:57.000 They're tireless compared to us.

02:12:59.000 Like, they have different view of the world and can solve problems potentially in interesting ways.

02:13:04.000 So, yeah, like, there's tons and tons of positive value there.

02:13:08.000 And even that we've already seen, right?

02:13:09.000 Like, past performance, man.

02:13:11.000 Like, I'm almost tired of using the phrase, like, just in the last month because this keeps happening.

02:13:17.000 But...

02:13:17.000 In the last month, so Google DeepMind and Isomorphic Labs, because they're working together on this, but they came out with AlphaFold3.

02:13:25.000 So AlphaFold2 was the first...

02:13:28.000 Let me take a step back.

02:13:29.000 There's this really critical problem in molecular biology where You have proteins, which are just a sequence of building blocks.

02:13:37.000 The building blocks are called amino acids.

02:13:38.000 And each of the amino acids, they have different structures.

02:13:41.000 And so once you finish stringing them together, they'll naturally kind of fold together in some interesting shape.

02:13:46.000 And that shape gives that overall protein its function.

02:13:50.000 So if you can predict the shape, the structure of a protein based on its amino acid sequence, you can start to do shit like design new drugs.

02:13:59.000 You can solve all kinds of problems.

02:14:01.000 Like this is like the expensive crown jewel problem of the field.

02:14:07.000 AlphaFold2 in one swoop was like, oh, like we can we can solve this problem basically much better than a lot of even empirical methods.

02:14:17.000 Now AlphaFold3 comes out.

02:14:19.000 They're like, yeah, and now we can do it if we tack on a bunch of...

02:14:22.000 Yeah, there it is.

02:14:23.000 If we can tack on a bunch...

02:14:24.000 Look at this quote.

02:14:27.000 AlphaFold3 predicts the structure and interactions of all of life's molecules.

02:14:34.000 What in the fuck, kids?

02:14:35.000 Of course.

02:14:37.000 Introducing AlphaFold3, a new AI model developed by Google DeepMind and isomorphic labs by accurately predicting the structure of proteins, DNA, RNA, ligands,

02:14:52.000 and more, and how they interact.

02:14:56.000 We hope it will transform our understanding of the biological world and drug discovery.

02:15:03.000 So this is like just your typical Wednesday in the world of AI, right?

02:15:07.000 Because it's happening so quickly.

02:15:08.000 Yeah, that's it.

02:15:09.000 So it's like, oh yeah, another revolution happened this month.

02:15:12.000 And it's all happening so fast and our timeline is so flooded with data that everyone's kind of unaware of the pace of it all, that it's happening at such a strange exponential rate.

02:15:23.000 For better and for worse, right?

02:15:25.000 And this is definitely on the better side of the equation.

02:15:28.000 There's a bunch of stuff like...

02:15:29.000 One of the papers that actually Google DeepMind came out with earlier in the year was in a single advance, like a single paper, a single AI model they built, They expanded the set of stable materials.

02:15:44.000 Coffee's terrible.

02:15:45.000 I'll just tell you right now.

02:15:46.000 Jamie, it sucks.

02:15:47.000 Poor Jamie.

02:15:48.000 I love terrible coffee.

02:15:48.000 The water never got hot.

02:15:52.000 Yeah, that's what it is.

02:15:53.000 It just never, never really brewed.

02:15:55.000 It's terrible.

02:15:55.000 Terrible coffee's my favorite.

02:15:56.000 Yeah, I can solve that problem too, probably.

02:15:58.000 Wait till you try this with terrible coffee, though.

02:15:59.000 You're gonna be like, this is fucking terrible.

02:16:01.000 Let me go get some cold ones.

02:16:02.000 Bullshit.

02:16:03.000 It's terrible.

02:16:04.000 It's terrible.

02:16:05.000 Yeah, I can just see that calculation of like...

02:16:07.000 Like if you're dating a really hot girl and she cooks for you?

02:16:09.000 Like, thank you.

02:16:11.000 This is amazing.

02:16:12.000 This is the best macaroni and cheese ever.

02:16:15.000 In fairness, if Scarlett Johansson's voice was actually giving you that copy.

02:16:20.000 Oh, I feel like this is the best copy I've ever had.

02:16:22.000 Keep talking.

02:16:23.000 May I have some more, please, governor?

02:16:26.000 Yeah, so there's this one paper that came out, and they're like, hey, by the way, we've increased the set of stable materials known to humanity by a factor of 10. Oh, my God.

02:16:37.000 So, like, if on Monday we knew about, you know, 100,000 stable materials, we now know about a million.

02:16:43.000 They were then validated, replicated by Berkeley University, or a bunch of them, as a proof of concept.

02:16:47.000 And this is from, like, you know, the stable materials we knew before, like, that Wednesday were from ancient times.

02:16:53.000 Like, the ancient Greeks, like, discovered some shit.

02:16:55.000 The Romans discovered some shit.

02:16:57.000 The Middle Ages discovered...

02:16:58.000 And then it's like, oh, yeah, yeah, all that?

02:17:00.000 That was really cute.

02:17:01.000 Like, boom.

02:17:04.000 Instantly.

02:17:04.000 One step.

02:17:06.000 And that's amazing.

02:17:07.000 We should be celebrating that.

02:17:09.000 We're going to have great phones in 10 years.

02:17:10.000 Dude, we'll be able to get addicted to feeds that we haven't even thought of.

02:17:15.000 So, I mean, you're making me feel a little more positive.

02:17:19.000 Like, overall, there's going to be so many beneficial aspects to AI. Oh, yeah.

02:17:23.000 And what it is is just...

02:17:27.000 It's an unbelievably transformative event that we're living through.

02:17:30.000 It's power, and power can be good and it can be bad.

02:17:35.000 Yeah, an immense power can be immensely good or immensely bad.

02:17:38.000 And we're just in this, who knows?

02:17:41.000 We just need to structurally set ourselves up so that we can reap the benefits and mine the downside risk.

02:17:47.000 Like, that's what it's always about, but the regulatory story has to unfold that way.

02:17:50.000 Well, I'm really glad that you guys have the ethics to get out ahead of this and to talk about it with so many people and to really blare this message out.

02:17:59.000 Because I don't think...

02:18:00.000 There's a lot of people that, like, I had Marc Andreessen on, who's brilliant, but he's like, all in.

02:18:06.000 It's gonna be great.

02:18:07.000 And maybe he's right.

02:18:09.000 Maybe he's right.

02:18:09.000 Yeah.

02:18:10.000 Yeah.

02:18:10.000 But I mean, you have to hear all the different perspectives.

02:18:12.000 And I mean, like massive, massive props honestly go out to the team at the State Department that we work with.

02:18:19.000 One of the things also is over the course of the investigation, the way it was structured was it wasn't like a contract and they farmed it out and we went out.

02:18:28.000 It was the two teams actually like work together.

02:18:31.000 The two teams together, the State Department and us, we went to London, UK. We talked and sat down with DeepMind.

02:18:38.000 We went to San Francisco.

02:18:39.000 We sat down with Sam Altman and his policy team.

02:18:42.000 We sat down with Anthropic, all of us together.

02:18:44.000 One of the major reasons why we were able to publish so much of the whistleblower stuff is that those very individuals were in the rooms with us when we found out this shit and they were like, Oh, fuck.

02:18:58.000 Like, the world needs to know about this.

02:19:01.000 And so they were pushing internally for a lot of this stuff to come out that otherwise would not.

02:19:05.000 And I also got to say, like, I just want to memorialize this, too.

02:19:09.000 That investigation, when we went around the world, we were working with some of the most elite people in the government that I would not have guessed existed.

02:19:20.000 Yeah.

02:19:20.000 That was honestly more.

02:19:23.000 Well, I can be...

02:19:24.000 It's hard to be specific.

02:19:26.000 Did you see any UFOs?

02:19:27.000 Tell me the truth.

02:19:28.000 Did they take you to the hangar?

02:19:30.000 No, no, there's no hangar.

02:19:32.000 Yeah, I believe it.

02:19:33.000 Can we cut that?

02:19:35.000 There's no hangar, Joe.

02:19:36.000 Don't worry, sweetie.

02:19:37.000 Don't worry about it.

02:19:44.000 We didn't go that far down the rabbit hole.

02:19:48.000 We went pretty far down the rabbit hole.

02:19:50.000 And yeah, there are individuals who are just absolutely elite.

02:19:55.000 The level of capability, the amount that our teams gelled together at certain points, the stakes, the stuff we did, the stuff they made happen for us in terms of bringing together...

02:20:12.000 Yeah.

02:20:28.000 And it was wild.

02:20:29.000 I mean, again, it's like— Trevor Burrus That was in November.

02:20:31.000 Aaron Ross Powell It's us two frigging yahoos.

02:20:32.000 Like, what the hell do we know and our amazing team?

02:20:35.000 And it was, yeah, referred to by—there was a senior White House rep there.

02:20:40.000 It was like, yeah, this is a watershed moment in U.S. history.

02:20:42.000 Trevor Burrus Wow.

02:20:43.000 Well, that's encouraging because, again, people do like to look at the government as the DMV. Aaron Ross Powell Yeah.

02:20:47.000 Trevor Burrus Or the worst aspects of bureaucracy.

02:20:50.000 There's missing room for things like congressional hearings on these whistleblower events.

02:20:55.000 Certainly congressional hearings that we talked about on the idea of liability and licensing and what regulatory agencies we need just to kind of start to get to the meat on the bone on this issue.

02:21:04.000 But yeah, opening this up I think is really important.

02:21:08.000 Well, shout out to the part of the government that's good.

02:21:11.000 Shout out to the government that gets it that's competent and awesome.

02:21:14.000 And shout out to you guys because this is heady stuff.

02:21:19.000 It's very difficult to grasp.

02:21:21.000 Even in having this conversation with you, I still don't know how to feel about it.

02:21:25.000 I think I'm at least slightly optimistic that the potential benefits are going to be huge.

02:21:31.000 But what a weird passage we're about to enter into.

02:21:35.000 It's the unknown.

02:21:36.000 Yeah, truly.

02:21:37.000 Thank you, gentlemen.

02:21:38.000 Really appreciate your time.

02:21:39.000 Appreciate what you're doing.

02:21:40.000 Thank you.

02:21:41.000 If people want to know more, where should they go?

02:21:43.000 What should they follow?

02:21:45.000 I guess gladstone.ai slash action plan is one that has our action plan information.

02:21:50.000 Gladstone.ai.

02:21:51.000 All our stuff is there.

02:21:52.000 I should mention, too, I have this little podcast called Last Week in AI. We cover sort of last week's events, and it's all about the sort of lens of...

02:22:01.000 You don't have to do that every hour.

02:22:02.000 Yeah.

02:22:03.000 Last hour in AI. A week is not enough time.

02:22:07.000 We could be at war.

02:22:08.000 Our, like, list of stories keeps getting longer.

02:22:11.000 Yeah, anything could happen.

02:22:12.000 Time travel.

02:22:12.000 You'll hear it there first.

02:22:14.000 Yeah.

02:22:14.000 All right.

02:22:15.000 Well, thank you, guys.

02:22:15.000 Thank you very much.

02:22:16.000 Appreciate it.

02:22:16.000 Bye, everybody.

The Joe Rogan Experience - May 25, 2024

Joe Rogan Experience #2156 - Jeremie & Edouard Harris

Episode Stats

Length

Words per Minute

Word Count

Sentence Count

Misogynist Sentences

Hate Speech Sentences

Summary

Transcript