Jim Keller is a microprocessor engineer known for his work at Digital Equipment, AMD, Apple, Tesla, and Intel. He was co-architect for what were among the earliest of 64-bit microprocessors, the EV5 and EV6, designed in the 90s. In the later 90s, he served as lead architect for the AMD K8 microarchitecture, including the original Athlon 64, and was involved in designing the Athlon K7 and Apple A4 through 7 processors. He is presently President and CTO at TENS Torrent, building AI computers. Dr. Jordan Peterson has created a new series that could be a lifeline for those battling depression and anxiety. We know how isolating and overwhelming these conditions can be, and we wanted to take a moment to reach out to those listening who may be struggling. With decades of experience helping patients, Dr. Peterson offers a unique understanding of why you might be feeling this way. In his new series, he provides a roadmap towards healing, showing that while the journey isn t easy, it s absolutely possible to find your way forward. If you re suffering, please know you are not alone. There s hope, and there s a path to feeling better. Go to Dailywire Plus now and start watching Dr. J.B. Peterson on Depression and Anxiety. Let s take the first step towards the brighter future you deserve. Today. - Dr. B. Peterson - Daily Wire Plus - Podcast Episodes: Today's Theme Song: "Incomptech" by Suneaters by Haley Shaw Music by Jeff Kaale (Recorded in Los Angeles, CA - "Goodbye Outer Space" by Cairo Braga (ft. & Other Words) & "Outer Space Warning" by Robert Ferendle (feat. ) by Jeff McElroy (Partially Accompanied by and "Let's Talk About It (Goodbye, Goodbye, Goodbye, My Love & Goodbye, & " by Squeepee) - "Solo" by F&R (Isaac) by Eddy (Solo) & (Feat. (Aptest) (Bennie) and (Amberly ( ) & ) ( ) ( ) ( ) & (Apostle) & (C) (Avenance ( ) )
00:00:00.960Hey everyone, real quick before you skip, I want to talk to you about something serious and important.
00:00:06.480Dr. Jordan Peterson has created a new series that could be a lifeline for those battling depression and anxiety.
00:00:12.740We know how isolating and overwhelming these conditions can be, and we wanted to take a moment to reach out to those listening who may be struggling.
00:00:20.100With decades of experience helping patients, Dr. Peterson offers a unique understanding of why you might be feeling this way in his new series.
00:00:27.420He provides a roadmap towards healing, showing that while the journey isn't easy, it's absolutely possible to find your way forward.
00:00:35.360If you're suffering, please know you are not alone. There's hope, and there's a path to feeling better.
00:00:41.780Go to Daily Wire Plus now and start watching Dr. Jordan B. Peterson on depression and anxiety.
00:00:47.460Let this be the first step towards the brighter future you deserve.
00:00:57.420Hello everyone, I'm pleased to welcome Jim Keller to my YouTube channel and podcast today.
00:01:16.000Jim is a microprocessor engineer known for his work at Digital Equipment, AMD, Apple, Tesla, and Intel.
00:01:27.420He was co-architect for what were among the earliest of 64-bit microprocessors, the EV5 and EV6 digital alpha processors designed in the 90s.
00:01:40.020In the later 90s, he served as lead architect for the AMD K8 microarchitecture, including the original Athlon 64,
00:01:49.360and was involved in designing the Athlon K7 and Apple A4 through 7 processors.
00:01:55.860He was also the co-author of the specifications for the X86-64 instruction set and HyperTransport Interconnect.
00:02:05.740From 2012 to 2015, he returned to AMD to work on the AMD K12 and Zen microarchitectures.
00:02:14.200At Tesla, he worked on automotive autopilot hardware and software, designing the Hardware 3 autopilot chip.
00:02:21.440He then served as senior VP of silicon engineering, heading a team of 10,000 people at Intel.
00:02:31.440He is presently president and CTO at TENS Torrent, building AI computers.
00:02:40.020He's also my brother-in-law, and we've talked a lot over the last 20 years.
00:02:50.020He was a friend of mine before he married my sister, and we've known each other for a very long time since we both lived in Boston.
00:02:56.180So I'm very happy to have you to talk to you today, Jim.
00:03:26.460Well, the long story is I graduated college as an electrical engineer with a bachelor's, and I took a job in Florida because I wanted to live on the beach.
00:03:36.920And it turned out to be a really interesting job.
00:03:39.360I was at Harris, and I spent like two years working in labs, fixing up electrical equipment and doing some networking stuff and some digital design.
00:03:48.240And at some point, a friend told me I should work at digital.
00:03:50.920So I read about digital, and I literally read the computer architecture manual for the VAC 780 on the plane on the way to the interview.
00:03:59.860And then I interviewed them with a whole bunch of questions because I just read this architecture spec, which I didn't know that much about, to be honest.
00:04:07.240But I was kind of a wise-ass as a kid.
00:04:10.340And they hired me because they thought I was funny.
00:04:12.860And so I got my architecture education working on the VAC 8800, working for a guy named Bob Stewart, along with some other really great architects.
00:04:23.480So I spent about seven years in that group, you know, learning to be a processor architect.
00:04:29.860And then I spent a little time at digital research in California for six months, and then went back and joined the Hudson team where they were building alpha processors and became co-architect with Pete Bannon on EV5 and then with Dirk Meyer on EV6.
00:04:44.920Now, those chips, 15 years, and we worked on, I would say, three very successful shipping products, and then a couple other products that didn't get to market.
00:04:58.740So those chips, from what I remember, were remarkably ahead of their time, but that didn't seem to save Digital Equipment Corporation.
00:05:49.580Well, we're going to return to the topic of crazy things going on inside computers.
00:05:52.880But it's interesting to note right there.
00:05:54.720So that's a situation where a company has a great product but doesn't know how to launch it into the marketplace or is blinded by its own preconceptions.
00:06:03.280It can't even say necessarily what it happened.
00:06:04.840More than that, Gordon Bell was CTO, and he was a really brilliant computer architect, but he also had really good, let's say, observational skills.
00:06:13.160And midway through the Vax 8800, he decided the technology we were using was a little too late and redirected the program.
00:06:21.260And it became a very successful product because he knew what was going on and he made decisions like that.
00:06:26.140But when he left, I think digital became an argument between business unit managers, not about technology.
00:06:34.460And Alpha was a great technology, but it went into business units that were aiming at high prices and high margins, not market penetration, and not basically keeping up with the software revolution that was happening at the time.
00:06:47.580And so it was Ken Olson was a great manager, but he wasn't a technical leader.
00:06:53.720And without Gordon Bell, the company kind of lost its way.
00:06:57.880And when companies lose their way, they fail.
00:07:31.580And that's not a tremendously long period of time.
00:07:35.380So there's always dominant companies and always a handful of dominant companies, but the company that's dominant tends to shift quite quickly.
00:07:43.060Well, I should return to this a couple of times.
00:07:45.520There's the, you know, the classic escrow in economics.
00:07:48.580You start out low, you solve a problem, you ramp up, you plateau, and then you fail.
00:07:53.260And this dominates business, it dominates humanity at some level.
00:07:58.120And it, you know, plays out over and over.
00:07:59.860Now, it sounds like you didn't have, so how is it that you managed to do this job?
00:08:06.000You intimated when we were talking that you weren't really trained for it.
00:08:09.540And so you were trained as an engineer.
00:08:11.780You had an, is it a bachelor's degree in engineering?
00:08:15.080And how prepared were you as a consequence of your degree for any of the jobs that you undertook?
00:08:22.880So, so a good engineering degree is math, physics, basic understanding of science, and some smattering of communication skills.
00:08:31.220You can probably do a great engineering degree in two or three years if you're dedicated to it.
00:08:36.780You know, the, the things that stretch my brains the most when I went to college is with math and mechanical engineering.
00:08:41.940And Penn State, I went to Penn State, and they used mechanical engineering and math as two of the weed-out courses to find out if you had the chops or the gumption to get through engineering.
00:08:54.980So there was a fairly high failure rate there.
00:08:58.000But mechanical engineering is a really interesting discipline because you have to think, think about solving math problems spatially.
00:09:04.220Like, you know, do things like, how do you calculate the force on a rotating, accelerating object?
00:09:16.360And in engineering school, you never answer a multiple choice question.
00:09:20.180You learn stuff and film is in stuff, but then you calculate and to understand what the result is.
00:09:27.800And the problem sets are like little design exercises.
00:09:31.540So how much of it do you think, how much of it do you think is pure screening, let's say, well, it wouldn't be pure screening, but fundamental screening for conscientiousness and IQ and how much of it is learning to think.
00:09:48.580How much of the education process is that?
00:09:50.900Like, if you're going to hire an engineer, are you hiring fundamentally on the basis of IQ and you get smarter people from the top schools?
00:09:57.100Or do you think that the engineering training actually does prepare people for a technical career?
00:10:03.780Well, it depends on the engineer and it depends on the school and depends on their approach.
00:10:08.380So my IQ isn't super high compared to really smart people.
00:10:33.680So I had to learn how to do the work, go through the mechanisms, automatize some of the basics so I didn't have to think about them so hard.
00:10:42.780But literally let my brain work on this stuff so that I could use them to go problem solve.
00:10:49.680And especially in engineering, there's lots of different kinds of engineering.
00:10:52.780There's like highly technical stuff where you turn the crank, like a skilled lawyer might.
00:10:57.880But there's other stuff where you have to be really creative.
00:11:00.380You have an unsolved problem that nobody solved before.
00:11:03.080And as an engineer, you have a skill set, right?
00:12:44.520So you, you know, go build a new bridge that's never been built before.
00:12:48.300It's not like bridges are unsolved problems.
00:12:50.700This particular bridge hasn't been solved before.
00:12:53.140You know, maybe unique challenges to it.
00:12:55.460But it's not like physics where you're looking for an unknown particle or, you know, it's, you know, there's a pretty big dividing line between engineering and pure science.
00:13:05.340Engineers typically work in domains where there's many, many knowns and the unknowns are problems of the combination of, you know, reality, you know, complexity.
00:13:15.720Whereas physics, physics in principle, they're working on stuff that's fundamentally unknown.
00:13:21.720As soon as it's known, they have to move on because, because then it's engineering, like, like physicists, you know, translate the unknown into engineering and engineering applies known concepts to unknown problems.
00:14:15.340So those were 64 bit chips that you guys designed to compete with the, the, the Intel chips that had dominated the, the, uh, home computer market at that point.
00:14:25.740Well, so there's a funny thing, which is.
00:14:27.700Like at some level building fast computers, isn't that hard, right?
00:15:04.040So, so every design bill has kind of a, you know, a range that it can play.
00:15:10.000And, and, and, you know, you, you, you build the first one and you know, you can make some improvements, but some point the improvements don't really help that much.
00:15:33.300The K5 was their first design and it didn't work out that good.
00:15:36.720And then they bought a company called Next Gen, which had K6, which is an okay design, but it wasn't competitive against Intel.
00:15:42.840And then K7, Dirk was the chief architect of, and he designed a computer that was competitive and the head of Intel from, and some of that came from our work at digital on UV5 and UV6.
00:15:58.360And some of it was just saying in this, this day, you know, we have this many transistors, but you get more transistors every generation.
00:16:06.860So you can basically imagine you're building a house, suddenly you have way more bricks and way bigger steel beams.
00:16:12.040So your idea about what to build has to scale with that.
00:16:17.320And then K7 was a 32-bit chip and then K8 was a 64-bit chip, you know, somewhat related to that as it turned out, but also it was built to be bigger.
00:16:28.400And what I did is I wrote the performance model.
00:16:34.080I came up with the basic architecture and I started to organize the team around building it.
00:16:38.420And while we were doing that, we also wrote the thing called hypertransport spec, which became the basis of essentially all modern server computers or what's called two-socket servers.
00:16:48.420We wrote that in 98 and in 2002 or 2022, they're still building them that way.
00:16:54.540And when you say you wrote it, what does that mean?
00:16:56.820What does the process of writing that entail?
00:16:59.900What is it that you're writing and how do you do that?
00:17:14.420And there's a couple of pictures and, you know, computer protocols are pretty straightforward.
00:17:18.620There's a command, there's the address you're talking to, there's the data you're moving, there's some protocol bits that tell you how to exchange commands, right?
00:17:25.920And then Dirk took the spec and said, you mind if I flesh this out a little bit?
00:17:30.280And three days later, he sent me a 50-page version of it, which clarified all the little bullets.
00:17:35.440And then that specification, we literally used to build the interface between K8s, right?
00:17:52.040It's very difficult for non-engineers to understand any of this.
00:17:55.780AMD market share and server went from 0% to 35%, which was a huge impact to the business.
00:18:04.320And it became essentially the standard because apparently Intel had a version of that, but it didn't go to market.
00:18:10.840But after Opturon came out to market, Intel built a similar version, similar protocol about how to connect a small number of processors together with that kind of interconnect.
00:18:21.840And then that, let's say, design framework became standard in the industry.
00:18:27.060So if you go into a Google data center and you pull it out, there'll be two sockets with an interconnect between the two of them.
00:18:34.360And each socket will have memory attached to it.
00:18:36.440And they call it the 2P server or two processor server.
00:19:19.760When we built them, the big server guys, servers used to be backplanes like this big with multiple CPU slots, multiple memory cards, multiple I.O. slots.
00:19:29.380And the server manufacturer thought the server is oriented around the back.
00:19:33.780So IBM, HP, Dell, they all turned this down.
00:19:36.660But all the little startups at the time, like Google, were using PCs as low-cost servers.
00:19:43.900And we made this, basically, you could take a PC board, instead of putting one computer on it, you could put two, which radically saved the money.
00:19:51.700So when AMD made those kinds of servers, it was a way lower entry point for server-class technology.
00:19:57.680And the little startups at the time used it, and then over 15 years, disrupted all the big server manufacturers.
00:20:06.480So it's, you know, it's one of those, I couldn't say we planned it.
00:20:11.080Like, the constraints that we had a target market, we didn't know that it was going to become essentially how servers were built, you know, for 20-odd years.
00:21:21.360And he had a group of architects that looked at what Apple was doing and figured out what they should do next.
00:21:26.960And I worked on a MacBook Air definition, like I wrote the power management spec and did some other architectural work, which ultimately was an NVIDIA chip called MCP89.
00:21:40.160And then I was one of the chief architects of, you know, four generations of SOCs, what's called A4, A5, A6, A7.
00:21:48.820And we did a lot of stuff there, but the division was, you know, mobile phones.
00:21:58.780To pack a computer into a phone, you have a piece of silicon about that big.
00:22:03.380And all the components, the CPU, the GPU, the IO, are all on the same chip.
00:22:08.420And when they first started building phone chips, they were considered to be very slow, low cost, you know, very integrated chips.
00:22:17.220And we thought, if you looked ahead, because technology shrinks about every two years, and about six or eight years, we'd have enough transistors on a phone chip, that would be more powerful than a PC at the time.
00:22:29.280So we started architecting computers, interconnects, and other functions, so that when we had enough transistors, we could literally have, you know, a high-end desktop in a phone.
00:22:43.480And Apple's DNA is, you create the product that kills your current product.
00:31:46.320Like, imagine that the creative process has a productive component and then a culling component.
00:31:52.040And the productive component looks like it's associated with openness.
00:31:55.320But what the culling component is, is open to question.
00:31:59.300And it does seem to me that, at least upon occasion, it's low agreeableness.
00:32:04.100It's the ability to say, no, we're going to dispense with that and to not let anything stand in the face of that decision, which would also include often human compassion.
00:32:18.300And people have different approaches to it.
00:32:19.880Like Jobs would call things those they weren't beautiful or they weren't great.
00:32:23.140But, you know, Elon Musk is famous for getting the first principles and really understanding it fundamentally and culling from like a standpoint of knowledge.
00:32:33.760And you've asked me, like, what makes an engineer great?
00:32:37.200Like, so you have to have the will to creativity.
00:32:40.580Like, now there's lots of engineering jobs that aren't creative.
00:32:43.420Like, you need a skill set, you can exercise the skill set.
00:32:46.340But if you're going to build new things, you need to be creative.
00:32:48.980But you also have to have a filter good enough to figure out what's actually good and bad.
00:32:55.100Like, I know a lot of really creative engineers and they find a new thing and they're excited and they go down the rabbit hole on it and they, you know, they can work on it for six months and nothing to show for it.
00:33:03.800So you have to have that conscientiousness.
00:33:06.160I don't know if it's conscientiousness, disagreeableness, you know, that, that taste on how.
00:33:10.760Well, the conscientiousness would keep you working in the direction that you've chosen and, and doing that diligently and orderly.
00:33:18.780The, the low agreeableness, well, that's, that's the open question because agreeableness is such a complicated dimension.
00:33:25.100There's obvious disadvantages and advantages at every point on the distribution.
00:33:29.780I mean, disagreeable people are often harder to work with because they don't care much about your feelings.
00:33:34.460But one thing I've noted about working with disagreeable people is you always know what they're thinking.
00:33:39.360And if you want someone to tell you what's stupid and wrong, they're perfectly willing to do that.
00:33:54.720Like he was very unemotional about it.
00:33:56.680Like he goes, Jim, I really like this and this, but this isn't working for shit.
00:34:00.840Like, what are we going to do about it?
00:34:02.000And, and you just, it would just be all shocked as a matter of fact.
00:34:06.200I would say when I was younger, I was a lot less disagreeable.
00:34:10.520You know, I'm fairly open minded and, you know, I like to create new stuff, kind of stuff, things.
00:34:17.380But then I saw enough things fail over the years because we didn't make the, you know, let's say the hard choices about something.
00:34:23.540And then, you know, you hate to work on something for two years and have it go away because at some point you realize you're doing a couple of wrong things.
00:34:31.060And you didn't do something about it when you could.
00:34:35.240And so as a, you know, as a manager and a senior leader, I'm somewhat famously disagreeable.
00:34:41.940Part of it's an act to get people to move.
00:34:44.680And part of it's, you know, my beliefs that I can't have people dedicate themselves to doing bad things for very long because it'll, it'll bite us.
00:34:53.980Yeah, well, we've talked a little bit about this too, about the moral dilemma between agreeableness and conscientiousness.
00:35:03.020Agreeableness seems to me to govern short-term intimate relationships like that between a mother and a child.
00:35:08.540And it involves very careful attention to the emotional reactions of another person and, and, and the optimization of those in the short term.
00:35:18.580But conscientiousness looks like a longer term virtue and they come into conflict at some point because sometimes.
00:35:25.380They come into conflict in the midterm.
00:35:29.260It's, you know, it could just be, you know, how our brains see the future, but, but it's like, you know, if you're managing the group and you have to fire somebody, it's hard.
00:35:38.060But do you want to fire five people now or everybody later?
00:35:40.660Like, well, like once you've internalized that and taken responsibility for that decision, then making, you know, management, leadership, position choices is always hard, but it's so much better to make them.
00:35:54.100And then, then succeed than it is to fail because you couldn't make the hard calls.
00:36:00.400Who's, who's got the upper hand, you know, someone who fires early out of necessity, but is accurate and looking carefully or someone who, you know, is willing to let people drag on.
00:36:13.320I'll give you, I'll give you two counter examples of that.
00:36:15.400So Jack Walsh in his book, straight from the gut, a weird thing.
00:36:19.280He said, you know, once you have a doubt on somebody, you never act fast enough, which, you know, it took me years to really believe that.
00:36:26.420And then the other weird one is people say, Hey, I have this organization of a hundred people and there's five, five people that aren't working out, but I'm not sure who they are.
00:36:36.120So I'm going to be really careful because I don't want to accidentally fire a good person.
00:36:55.360You're better firing too many than too few.
00:36:57.520And how did you come to terms with that emotionally?
00:37:01.860I mean, look, we have a mutual friend who fires people with quite great regularity and I've talked to him and he scores very high in disagreeableness.
00:37:09.640And I talked to him about firing, which he's done a lot of, and he was actually quite positive about it.
00:37:15.020He said, I don't fire anyone who I don't think is causing more trouble than preventing.
00:37:19.520And so by firing the person that I'm firing, I'm actually doing a very large number of people, including potentially that person a favor.
00:37:28.260It didn't bother him, but he was temperamentally wired that way, I would say.
00:37:32.580But I would say, you know, digital equipment went bankrupt because they had bad people who didn't fire.
00:37:39.640I've seen many groups fail because they couldn't clean house.
00:39:04.460So there's this idea that you fire the bottom 10% of a team because the random people you hired will be better on average than the bottom 10% of your team.
00:39:53.500Maybe you're ranking a little wrong, but impact on morale is high.
00:39:56.880Teams, generally speaking, Rory Reed was CEO of AMD when I joined and we had a big layoff, which we had to do because we were running out of money.
00:41:10.800It's hard stuff to do that creates something really great.
00:41:15.360Like when I joined AMD in, what was it, 2013 or something, like they had two product lines, you know, Bulldozer and Jaguar, and they're both failing.
00:41:39.400It's like, you know, we needed to be building eight bedroom houses and we're trying to add six bedrooms to a two-bedroom house in one case.
00:41:49.200And the other one was structurally screwed up.
00:41:51.160It seemed to be the right ballpark for the performance we should get, but the way it was engineered and built was sort of like, you know, you let the plumber do the architecting and the house looks like shit.
00:43:42.860So if you know you're going in the wrong direction, a change in direction can maybe is just as bad, but there's some chance it's good, especially if you're somewhat smart and you have some experience.
00:44:08.020And then the other problem is when you build a house, it has a foundation.
00:44:11.880Once the foundation is built, it's very difficult to change the top of the house a lot.
00:44:17.660Like if you have a foundation for a two-story house, it's hard to make it into an eight-story house.
00:44:22.500So when we cancel those projects, we consciously reset some design methodologies, some team organizations, some leadership, some let's say, you know, we said we're going to have the best in class leadership, design methodology, and some of the architectural tools.
00:44:44.400We're just going to take those as givens.
00:44:46.780Now that the land has been cleared, we had the opportunity to go back to that.
00:44:52.500And it was interesting in the design teams, it turns out there was some very good pieces, you know, in the two processors they had, but they weren't working together organically like they should.
00:45:03.880And let's say the framework of the design wasn't big enough.
00:45:07.460And then the tools over the years have evolved into lots of little local improvements, but it wasn't really the right tool set.
00:45:15.740Now, AMD leapt forward when you did this, and they were the only competitor to Intel in a realistic sense.
00:45:21.640And so this, these actions on your part were part of what made that company thrive and kept competition within the microprocessor world.
00:45:29.960So these didn't have, these decisions didn't have trivial outcomes.
00:45:35.700And there was, the really cool thing was, you know, when we did that, we didn't really bring in outsiders.
00:45:41.380Like that Zen design was entirely based on people who worked at AMD at the time, right?
00:45:47.760So what we needed was to clear the plate a little bit, to reestablish some, you know, first principles about how we were doing things to have a better goal.
00:45:57.700There was a little bit of head knocking on getting like the methodologies straightened out.
00:46:03.340You know, I was, let's say, fairly disagreeable about how we were going to get through that because people kept saying, oh, it's too hard to do this.
00:50:30.480And he likes the details and he has a good eye for it.
00:50:33.020Usually before, before Tesla, when you do a technical presentation, usually done, you have some kind of methodology for presenting the problem to an executive.
00:53:27.440Well, I've seen in the software projects, other projects that I've been involved in, too, that focusing on a solution, I think this is along the same lines as what you're discussing, is, well, then you get to product a lot faster.
00:53:39.660It's like, this thing has to exist and work.
00:53:42.220Maybe it won't solve, and the problem with the problem is that you can indefinitely investigate the problem and expand it, and also that that feels like work, but it's not saleable.
00:53:54.040Yeah, it puts in the forefront of your mind, you know, that constant, you know, you need to be creative in pursuing the problem, but also make sure you're really on track of the solution, and you're just not falling in love with the problems.
00:54:07.760Some engineers are great at admiring problems.
00:54:09.540Like, I've worked with lots of people that come in with a 12-page presentation, and when I'm done, I'm like, did you guys just give me a whole bunch more data on the problem?
00:54:18.020He's like, yeah, we're really getting to the bottom of it.
00:55:13.980But if you're in charge of solving problems, you know, that period needs to be focused, short, concise, and you need to move on to solutions, right?
00:55:24.680I think that's probably why I'm not so temperamentally fond of activists.
00:55:42.560Well, so a large number of people, and this is my favorite thing I learned, and it was working with mechanical engineers at Tesla, because they think the world's made out of silly putty, right?
00:55:53.760They used to design, when we were building Model 3, they would design the part, and then they would joke about how they're going to make it.
00:55:58.920Are they going to, you know, CNC it, like mill it?
00:56:01.480Are they going to injection mold it, 3D print it, stamp it, make it with a hammer, you know, cut it out with scissors, you know, carve it out of a block?
00:56:09.540Like, they had this cool machine that could carve 3D models out of clay.
00:56:16.200Like, so they could design things in their heads and on computers, and then go build any damn thing they want.
00:56:22.380Like, if you ever look at a complicated mechanical assembly, there would be some screwed aluminum thing that would be milled somewhere and then drilled, and there's screws going through it.
00:56:32.400And there'd be some little tab sticking off of it that holds another thing.
00:56:38.820They think they can make anything, right?
00:56:42.400And there's a whole bunch of people in the world that don't think they can make anything.
00:56:44.960They don't, they think the world is what it is.
00:56:49.000I had a friend, he had a rattle on his dashboard, and he didn't know what to do about it.
00:56:55.540And I was asking him where the rattle was, and, you know, and I was thinking that, I was talking to him about, like, how the dashboard's made.
00:57:05.940You think the dashboard's made of a whole bunch of parts that are put together some way.
00:57:10.540I thought the dashboard was just the dashboard.
00:57:12.620Like, he could, he didn't conceptualize it as there's this outer piece, and there's inner brackets, and there's radio, and there's just these things.
00:57:20.740Whereas mentally, I can't help but see the whole thing in 3D, and then I'm wondering which piece is loose and where it is, right?
00:57:47.080You know, for human evolution, like, you know, it was pretty much the same for a million years.
00:57:52.980Like, it's weird how good we are at change.
00:57:54.700And my best theory on it is from 0 to 20.
00:58:00.580Like, your brain is going through radical change because you're going from, you know, silly putty and not knowing much to being pretty smart.
00:58:07.300So you have to change and adapt really fast.
00:58:10.040And then humans are adapted to deal with each other, and humans are fairly crafty.
00:58:49.720Like, you never knew what they were up to.
00:58:51.620Like, one day, like, they were working on the interior for the car, and they made this crazy-looking model, which kind of looked like a car, but it turned out it was a thing you could move around and have the attachment points for all the interior parts.
00:59:03.580So you could basically, it looked like a weird skeleton, but it had the attachment points that you could adjust, and you could build all the interior parts and put a Tesla interior together right in the middle of the, where the engineering desks were.
00:59:28.960When you look at something that's small, even if you scale it up perfectly when it's big, sometimes that's just what you thought, and sometimes it doesn't work.
00:59:37.140And so, you know, you want to do, you know, computers like to change things fast, but like real scale models that you sit in and live in and, you know, get a human experience for it.
00:59:47.100And it was really fun for that to just show up and be like, holy cats.
00:59:50.120If I did a similar thing, we took all the electrical subsystems and motors and laid them out on two tables, two big tables, covered with all the electrical parts of a Model 3.
01:00:00.140We stared at it, and once you see them all together, it's crystal clear, that could be a lot better.
01:00:07.340Because, you know, there's three motors that look almost the same, why isn't that one motor?
01:00:11.520There's these two parts that are completely separate assemblies, but if you build it together, you could have one thing do both things really much more naturally.
01:00:19.380Right, and by laying that all out in front of you, you didn't have to do the mental work of representing that.
01:00:25.520You could do the mental work of seeing how all the parts interconnected and what might be.
01:01:13.700I really don't understand how computation works.
01:01:17.700And you're a microprocessor architect and you build computers.
01:01:20.720And I listened to a discussion you had earlier this month, and there was a lot of it I couldn't follow.
01:01:28.340I thought it might be helpful and interesting for you just to walk through for me and for my audience how a computer actually works, what it does, and how you build it, and then what it would be like to design and to architect a microprocessor.
01:01:45.620Yeah, well, it's somewhat hard to describe, but there's a couple simple things.
01:01:52.800So, let's start with the easiest thing.
01:01:54.640The computers have three components, really.
01:01:57.860Memory, programs, and input and output.
01:02:03.320Right, those are the three basic things we always build.
01:02:06.880Right, so memory is like the DRAM or the disk drive, place where you store data.
01:02:18.100So, you can take any bit of information and describe it as a sequence of ones and zeros.
01:02:25.940And it's stored in silicon in either static and dynamic memories or on disk drives, which, you know, there's a couple technologies for that.
01:02:37.680Well, it does, although I have some difficulty in understanding exactly how the transformation is undertaken to represent things in zeros and ones.
01:02:49.540So, if you shine a light on a photo, a photo detector, right?
01:02:55.760So, the light comes in and it's a stream of photons, right?
01:03:00.840And the photo detector counts the photons, literally.
01:03:05.760So, every photon that hits it or a couple photons hit it, they cause some electric charge to move.
01:03:11.080And that causes the circuit to wake up and say, I saw some photons.
01:03:14.280So, say you're trying to evaluate how strong that light is.
01:03:20.040So, it could be anywhere from nothing to super intense, right?
01:03:24.340And then you might say, well, let's put that in a range of numbers from zero to a thousand, right?
01:03:31.620So, and then you're starting to light on it.
01:03:34.440And so, you count the photons for, say, you know, a microsecond.
01:03:37.220And then you translate how many photons you counted into the number, right?
01:03:43.060So, and so, just imagine as the light varies up and down, the count, the number coming out of your photo detector is varying between zero and a thousand, right?
01:03:54.060And we use base 10, but you can translate that to binary, which is base two.
01:07:38.240And the address corresponds to the physical location, in some sense, to the physical location of the, of the.
01:07:44.000And how are the zero and ones represented in the memory?
01:07:47.760It's literally a voltage that's either high or low.
01:07:50.740And zero is using ground, which is zero volts.
01:07:53.240It's in modern, you know, DRAM cells probably stored at 1.1 volts.
01:07:59.340And, you know, in a DRAM cell, it's a capacitor that's holding electrons.
01:08:03.560So, basically, when you store the cell and either you drain all the electrons out, so it's zero volts, or you put a bunch of electrons in so that it holds a one volt.
01:08:14.060So, it's literally a number of electrons in there.
01:08:16.520There's a couple of ways to make memory cells.
01:08:18.180There's another way, which is called a bistable element, where you have what's called cross-couple inverters.
01:08:23.020But that's too complicated to explain.
01:08:26.480And then, the memories are usually built in rays.
01:08:30.620So, you take the number, and you say, I'll take the bottom half of the number and figure out which row it's in, and the top half of the number of which column it's in.
01:08:38.100And where the column and the row overlap, then I'll write my new data of a one and zero in that spot.
01:08:46.260So, if you look at a memory chip, you'll see this array of bits with little blocks on two edges.
01:08:54.520Usually, you know, one side's the row, one side's the column, and then the bottom is what they call the sense end, and you read it back out again.
01:09:01.820So, a memory process is you activate the row and column to a spot, which gives you the address of that bit.
01:09:08.220And then, you drive the bit in and charge up or discharge that cell, and then it holds them.
01:09:22.860Way back when, there was something called Chlor memory, where they had essentially the XY grid, and at each little place, there was a little magnetic bead, which when you put the current through, you put the current in the same direction, you could make it be north to south and the opposite direction, south to north.
01:09:42.300So, you basically remagnetize the little beads.
01:09:45.240So, there's lots of ways to make memory, but currently, the really dense memories are called dynamic memories, where you literally put charge in there.
01:09:53.660And then, there's fun stuff that happens, like flash cells, the cells got so small that the electrons, you know, from the quantum effects, tunnel out occasionally.
01:12:34.000You know, then there's what's called logical operators, and or not.
01:12:37.320Well, it's stunning to me, conceptually, thinking through this, that computers, which can produce whole worlds, in some sense, can do that as a consequence of zeros and ones and arithmetic operators.
01:13:01.160So, at some low level, you know, like I understand computers from atoms up to operating systems, which is, you know, fairly broad range, but there's lots of people who can do that.
01:13:11.900Yeah, and I understand them like the surface of the keyboard.
01:18:48.820Now, why can computers construct worlds?
01:18:54.680So I still remember when, so if you look at your screen, you know, the computer in front of you probably has two or four million pixels on it.
01:19:04.180And when they first started, you know, televisions, when they lit up screens, you know, they were scanning the little electron microscope, you know, electron beam across a phosphorescent surface and lighting and modulating the intensity of the electron gun to make the little phosphores brighter.
01:20:13.420So it's digitized in the sense it's discrete, I'd say, in the sense that each little pixel, you can see the little phosphores in the screen.
01:21:17.960Well, it turns out for a whole bunch of reasons, like if you want to make something look really good on the screen, so the world is relatively continuous, right?
01:21:28.700So if you look at it, there's all this light reflecting around.
01:21:33.460There's no little pixels in the surface of your table, right?
01:21:37.680To make a discrete grid look that way, you have to, you know, combine the colors.
01:21:44.120You have to do a whole bunch of stuff.
01:21:45.580You have to pretend you're shining lights on it.
01:21:47.860You have to, you know, like there's a reflection from one surface to the next one.
01:21:51.100And it turns out when you have thousands of instructions per pixel, you can start to make those pixels look realistic, right?
01:22:00.640The operations, when you go look in the pixel program, like it looks so beautiful, but you think that's incredible.
01:22:06.860But if you look in the pixel program, it's load the data into the register, add it to a number, test it against the number, subtract something.
01:22:14.340There's something called clipping, like make sure the pixel doesn't get brighter than this and dimmer than that.
01:23:10.120Input and output is typically the way to, you know, it depends on what you're doing.
01:23:14.660You might just send bets from one place to another, but it might also be, you could say input, you know, input and output in the computer and sensors are slightly different things.
01:23:22.480Like sensors, you know, turn analog real world signals into bits.
01:23:28.780And then programs basically transform the data in some way.
01:23:34.320And programs is basically seek, you know, operations like add subtract, divide, and then branches that either let you do loops or make decisions.
01:23:44.580And then the hardware to let you do subroutines to break the program into pieces.
01:24:10.800Yes and no, it's not a very good, you know, ones and zeros is better because then it's a, it's a mathematical representation, you know, a digital representation of an analog reality.
01:25:30.980And they just, they can operate at that level.
01:25:33.340Then there's another thing where you take multiple transistors together and you basically make what's called logic gates, which literally do the ands and ors and inversions.
01:25:46.560We call it, you know, the physical design library or something like that.
01:25:50.100And then people take those and they make them up into adders and subtractors and multipliers.
01:25:54.920This is a well understood Boolean mass.
01:25:57.660So how do you, how do you add two binary numbers?
01:26:00.580So you make those and then there's another abstraction layer that says, all right, I'm going to take multiple operation units and put them together to make, you know, part of the computer.
01:26:11.360And then you make, there's a bunch of those blocks and then that thing runs a program very simply.
01:26:17.260And there's a small number of people who write programs at the low level, but then there's people who use what's called libraries where they, you know, they're, they're doing some higher level program.
01:26:25.800And so they're going to do a matrix multiplying and do this and that, but they don't actually write that low level code.
01:26:30.640So there's, you know, there's a stack of abstractions.
01:26:33.840And when something gets too complicated, you split the abstraction layer into two things.
01:26:39.480There used to be when, when people wrote a program, there's a program called a compiler that translated your C program or Fortran program into the low level instructions.
01:26:49.820But it turns out there's too many languages up here and there's too many instructions here.
01:26:53.960So now they translate it from the high level language into an intermediate representation, which is sort of, let's say a generic program.
01:27:00.920And then there's another thing that translates the intermediate representation and the specific computer you have.
01:27:06.780But that just keeps going higher and higher.
01:27:08.940Like a lot of programmers, they, they use, you know, frameworks that can do amazing things.
01:27:15.800Like you could literally lay a program and it says, search the internet for a picture of a cat sword by color output to my printer.
01:27:24.160Like there's a language where that's a program.
01:27:33.560That's a really expensive, that's a really complicated program.
01:27:36.180So how much of the radical increase in computation power is a consequence of hardware transformation and how much of it is a consequence of the, the, the increasing density, let's say of these abstraction layers.
01:27:49.960Well, so this is, this is where, you know, there's a really creative tension or dynamic interplay.
01:27:55.560So when computers first started, they were so slow.
01:28:03.660And we've been going up the math hierarchy.
01:28:05.520So then you could run a program that did what's called, you know, matrix math, like, or linear algebra systems of big equations, and then matrices, and then more complicated ones.
01:28:15.640So as the computational power went up, you could dedicate more and more stuff to, you know, that kind of computation.
01:28:24.660And then similar thing happened on abstraction layers.
01:28:27.820Like it used to be, if you bought a million dollar computer, you hand wrote every line of code because you didn't want to waste time on the computer with like overhead.
01:28:38.280But today, you know, that million dollar computer costs 10 cents.
01:28:42.060You don't really care how many cycles you use, you know, parsing a cat video or something.
01:28:46.880And so the computation capacity, let the abstractions at the programming level increase a lot.
01:28:55.560So somebody had a graph about how many bytes does it take to store the letter A?
01:29:01.060Like, it used to be one, and then Word for Windows, it's like 10 kilobytes per letter.
01:29:10.480Because the letter has a font, it has a color, it has a shadow, you know, there's a whole bunch of, you know, and that's fine.
01:29:17.300Like, the computer with a million dollars for, you know, a thousand bytes of memory, you wouldn't store a letter A like that.
01:29:25.500But now you have gigabytes and terabytes of storage, who cares?
01:29:30.600You probably already know that there are data brokers out there selling your internet data off to companies who want to serve you a targeted ad.
01:29:37.720But you might be surprised to learn that they're also selling your information to the Department of Homeland Security and the IRS.
01:29:44.300Mask your digital footprint and protect yourself with ExpressVPN.
01:29:48.200One of the easiest ways for brokers to aggregate data and tie it back to you is through your device's unique IP address.
01:29:54.300But when you're connected to ExpressVPN, your IP address is hidden, making it much more difficult for data brokers to identify you.
01:30:02.020ExpressVPN also encrypts 100% of network traffic to keep your data safe from hackers on public Wi-Fi.
01:30:08.460You can download ExpressVPN on all your devices, your phone, your computer, even your home Wi-Fi router.
01:30:14.940Just tap one button and you're protected.
01:30:17.300Make sure your online activity and data is protected with the best VPN money can buy.
01:30:22.020Visit ExpressVPN.com slash Jordan right now and get three extra months free.
01:31:12.060So, and I hardly ever talk to the atom people, but I know something about atoms.
01:31:17.200So, I build, you know, when I'm architecting stuff, the functional units and then how they operate together at the low level that runs programs.
01:31:50.100And the computer would literally have that in it.
01:31:52.760So, that's an instruction, load the data, do the operation, which is a branch, execute the branch, if necessary, change the program counter.
01:32:03.360So, you know, people, you know, there was a period of time where computers had like five stages in it.
01:32:08.460And each one of them could say, that's the branch, that's the fetch unit, that's the load unit, that's the add unit, that's the branch unit.
01:32:15.640Right, but modern computers are more complicated than this, right, because computers like that would do one instruction every five cycles.
01:32:25.500And modern computers, the fastest one I know about, is doing 10 instructions, 10 instructions in a cycle in parallel.
01:32:38.200So, if you write a program, since you write, right, when you write, you write linear narratives.
01:32:46.520Right, you write a sentence that makes sense, followed by another sentence.
01:32:52.120Right, and so, as you're writing along, sometimes the one sentence defines the meaning of the next sentence.
01:32:57.960Right, and then group it into paragraphs.
01:33:01.260You might call those subroutines, right?
01:33:03.340And sometimes the paragraphs have to be ordered, and sometimes the paragraphs, the order doesn't matter.
01:33:09.740Right, so programs are written by human beings, and they're written in the same linear narrative.
01:33:15.640So, if you want to go faster than parsing the instructions one at a time, in order, you have to do some analysis to say, all right, I got two sentences.
01:33:28.280If they're dependent, I do them in order.
01:33:30.380If they're not dependent, I can do them in parallel, or any order.
01:33:33.760Right, and you start, so the modern computers, when they're reading the programs out, they're analyzing the dependencies and deciding what can happen in order.
01:33:44.460What has to happen in order for correctness, correct understanding, and what can be reordered.
01:33:49.700And then it turns out, there's many places where you say, if there's an error, go here, but there's hardly ever an error, and you can predict that really well.
01:34:01.260So, you say, I'm going, you're reading along, and you say, here's a point where I'm not sure which, should I read the next sentence, or should I jump to the next paragraph?
01:34:12.500Right, so a modern computer predicts that.
01:34:18.340It doesn't wait for you to fully understand all the sentences up to that point, so you know exactly where to read them to.
01:34:26.000So, imagine, so now you're, you're reading this book, and you're reading sentences in dependency order, which means you haven't, so you get to a branch, and you haven't read all the sentences before that and understood them.
01:34:37.540So, you don't know where to read the next paragraph, the next chapter, but we predict what's going to happen.
01:36:03.020And that's where you parse the instructions carefully, and you figure out what order you can do them in, and when possible, you reorder it.
01:36:32.420And that's because, and there's other sophisticated predictors in there.
01:36:36.320So, to do that, you have to fetch large groups of instructions at a time, you have to figure out where the, like, the sentence boundaries are, figure out if they're dependent or not, figure out if you can predict where the next instructions are coming from when you hit branches.
01:36:51.440And it turns out that's fairly complicated.
01:36:54.180The difference between a little computer that does, let's say, one instruction at a time and a complicated one that does 10 instructions at a time, it's 100 times more complicated.
01:38:02.220And there's a, there's a really good talk by Andre Carpathy called Software 2.0.
01:38:07.420So, all the, the first two kinds of computers, simple computers and complex out-of-order computers, they're running, running programs written by humans.
01:38:28.000Yeah, the different thing about AI computers is you use data to train the weights in neural networks to, to get you the desired result.
01:38:44.480So, instead of, the programs are no longer written by humans.
01:38:48.420Now, it turns out there's components of the AI stack that are written by humans, but at a high level, you use data to train them.
01:38:56.640So, you have a big neural network and you want to detect cats.
01:39:01.780So, you put a cat picture into the network when you start training and the output is gibberish.
01:39:07.060And you compare gibberish to what a cat is and you calculate the difference in what the network said versus the desired result, which is the word cat.
01:39:16.540And then they do something called backpropagation, which is mathematically sophisticated, but essentially takes the error and partition it across the layers of the network.
01:39:25.460Such that you've sort of bumped each neuron a little closer to saying cat next time by taking the bigger at the end and distributing it across.
01:39:35.460And then you put another cat in, and if you have the right size network and the right training methods, after you show that network a million cats, when you put a cat in, it reliably says cat.
01:39:48.540And when you put a picture that's not a cat, it reliably says it's not a cat.
01:40:35.520So, you could write code, and the problem with that is, well, now the cat has an arbitrary orientation.
01:40:40.780So, you have to, you do your feature detect on the picture, and the features have to search the whole image, and you have to rotate around.
01:40:48.100You know, and it's sort of, and every single thing you want to detect, you have to write a unique program for.
01:40:54.380You're done with cats, now you go to dogs.
01:40:56.720And then what about the dog that has slightly pointy ears?
01:40:59.740These dogs have round ears, and cats have pointy ears.
01:41:13.060I had a friend who worked on speech recognition years ago.
01:41:16.060So, you break speech into, you know, the phonemes, so you can see those, and then they have frequency characteristics, and you can differentiate vowels from consonants.
01:41:26.000So, those people working on speech were doing a whole bunch of analysis of analog waveforms of sound.
01:41:34.180And they were making some progress, but it never really worked.
01:41:38.800And then they train neural networks by, you put the word in, and you have what's called supervised learning.
01:41:47.940So, you play a language where you know what all the words are, and you keep telling the network how to correct.
01:41:54.100And with like a billion samples and a big enough neural network, it can recognize speech just fine.
01:42:00.360And if you train it with a broad variety of accents, it can work across accents.
01:42:10.040And then it turns out, the bigger they made these networks, the more information they can put in.
01:42:14.860And then, on the cat one specifically, they found, so, when they first had a neural network crack the cat problem, I forget, it was like 50 layers deep.
01:42:29.620And if you looked in the layers, you could see that it was detecting point of ears and eyes.
01:42:35.020But it was also detecting a lot of other things.
01:44:18.200You were an engineer and then you were a manager.
01:44:20.700And you've worked in lots of companies, some of which were incredibly creative, some of which were thriving to an incredible degree, and some of which were collapsing and irreparable.
01:44:30.660So, what have you learned about what makes companies work, and more importantly, what have you learned about what makes them not work, and maybe what do you do then?
01:46:09.720So then my skill set is somewhat unusual, and I'm not the only one, obviously.
01:46:14.340But I decided to, you know, get on the management track so I could build the computers I wanted.
01:46:20.860Because sometimes, when I wasn't the leader of the group, some managers, at some point, would decide they own the next decision,
01:46:27.760and they would make some random decision.
01:46:30.220I'd be grumpy about it, and there's nothing I could do about it, because people work for them, not for me.
01:46:34.860So that's, you know, it was a conscious thing.
01:46:37.840And I hired a consultant, Ben Katrao, who helped me reframe how I approached this.
01:46:43.740Now, I'm still a technical person, but I found that it turns out there's a whole bunch of really good technical managers that I like to work with.
01:47:26.880And there are lots of companies running away.
01:47:28.920A lot of times, founders tend to be technical people, but people working for them are non-technical.
01:47:33.200Or they're stronger on the management side than the technical side.
01:47:39.480But for everybody, you need to decide who you are.
01:47:42.860Like, I had a great technical manager at AMD, and one day, he was a little mess because I was looking to the, you know, a couple of the really technical heavyweights.
01:49:10.560And then the next was the organizational problem itself is an architectural problem.
01:49:16.920And then I kept, you know, for myself, well, I'm a funny kind of, if something has a solution and it's being confidently driven, I'm not that interested in it.
01:49:29.040And so in a big organization, there's a million problems and I start sorting them by priority and then solving some of them or handing them out to the right people.
01:49:38.640So there's a whole bunch of technical work to do on that.
01:49:42.040And then I'm fairly good at skill assessing people who are technical, either for management or technical positions.
01:50:24.340And do they have the, you know, is the organization that, like a lot of times, you know, there's a joke that startups start with a problem and build an organization and support it.
01:50:36.120But on the second, third system, the organization defines the problem rather than the problem defining the organization.
02:00:38.880That's a different, different problem.
02:00:41.140So, like, a company that's bureaucratically captured will manifest all kinds of bad behavior in the organization and product development.
02:00:49.400But, and then, you know, some big companies where the, you know, the bureaucracy has taken over, there might still be groups that are really doing a great job making great products.
02:01:00.480You know, so there's, you know, I think there are separate spaces and I understand both of them pretty well.
02:01:04.820So, and again, you know, the way you solve big complicated problems, you have some abstractions about what you're dealing with.
02:01:11.580So, you know, a framework like goals, organization, capability, and contract is, is a super clear method for evaluating what the hell is going on and then making changes.
02:01:20.900You know, very specific changes to it.
02:02:12.180So engineers need to have this will to create if they're technical leaders, let's say.
02:02:17.300And then they have to have the discernment to make, you know, decisions about whether they're actually making progress with the goals or just wasting their time on something cute.
02:02:49.940She would just have these team meetings and she would really get out there and energize the team.
02:02:55.180And another guy in the same building was very low key and he would wander around and talk to people and have a really good sense of the team, like a, like an introvert versus extrovert style.
02:03:06.640They were both very competent and they were both, to me, you know, really good technical competency.
02:03:13.880They weren't my technical leads, but they were technically competent enough to make the decisions and know when they had to punt the decision up.
02:03:21.860So, who do you not want, who do you not, okay.
02:03:25.720So, I mean, that kind of goes along with the management literature.
02:03:28.680You see that you want people who are intelligent, especially for complex jobs, so they can learn.
02:03:33.640You want people who are conscientious because they work hard and they have integrity.
02:03:37.480Then with the other dimensions, it looks like there's a fair bit of variability, although too much negative emotionality can be a problem.
02:03:44.680I think that's because it's associated with depression and too much anxiety and so on, but there's diversity in the other personality dimensions and that might be task specific.
02:03:54.240But what do you, what's, what, what sort of person do you not want to work with?
02:13:04.860Like, people think of college as expanding their possibilities.
02:13:08.040And the university itself has so many options, you think that would expand your possibilities.
02:13:12.460But once you pick one of them, and you study it for four, eight, ten years, you've narrowed your possibilities, right?
02:13:19.920You're kind of stuck with your discipline, and you pick that 20, which I think is crazy, by the way.
02:13:25.360Like, I think if you want to be an engineer, a good general engineering degree, like mechanical engineering or electrical engineering will give you thinking skill sets.
02:13:33.300I'm not a huge fan of people getting PhDs unless they really, really know they love it, right?
02:13:39.480And then take some jobs where, you know, there's an opportunity to do something for a year or two and then do something else.
02:13:47.400Like, I worked, my first job out of school was a random job.
02:13:51.440But I worked on, like, five different projects in two years while I was there, you know, fixing hardware, building something, debugging something.