The Peter Attia Drive - #143 - John Ioannidis, M.D., D.Sc.： Why most biomedical research is flawed, and how to improve it

00:00:00.000 Hey, everyone. Welcome to the drive podcast. I'm your host, Peter Atiyah. This podcast,

00:00:15.480 my website, and my weekly newsletter all focus on the goal of translating the science of longevity

00:00:19.800 into something accessible for everyone. Our goal is to provide the best content in health

00:00:24.600 and wellness, full stop. And we've assembled a great team of analysts to make this happen.

00:00:28.880 If you enjoy this podcast, we've created a membership program that brings you far more

00:00:33.280 in-depth content. If you want to take your knowledge of the space to the next level at

00:00:37.320 the end of this episode, I'll explain what those benefits are. Or if you want to learn more now,

00:00:41.720 head over to peteratiyahmd.com forward slash subscribe. Now, without further delay,

00:00:47.740 here's today's episode. I guess this week is John Ioannidis. John is by all estimates,

00:00:55.540 a polymath. He's a physician, scientist, a writer, and a Stanford University professor.

00:01:01.000 He has extensive training in mathematics, medicine, epidemiology. He's just generally one of the

00:01:06.940 smartest people I've ever met. And I've had the luxury of knowing John for probably about nine

00:01:11.440 years. And anytime I get to interact with him, whether it's over a meal or more formally through

00:01:16.340 various research collaborations, it's just always an incredible pleasure.

00:01:19.600 John studies scientific research itself, a process known as meta-research, primarily in clinical

00:01:26.720 medicine, but also somewhat in the social sciences. He's one of the world's foremost experts on the

00:01:32.300 credibility of medical research. He's the co-director of the Meta-research Innovation Center

00:01:37.980 at Stanford. In this episode, we talk about a lot of things. We talk about his journey from Greece

00:01:45.160 to the United States, but we talk a lot about some of his seminal papers. You're going to see me

00:01:50.480 reference a number of papers, beginning with, I think, one of the most famous papers he's written,

00:01:55.340 although by citation, it turns out to not be the most famous. There's actually papers that even

00:01:59.440 exceed it, which is an amazing paper where he describes through a mathematical model why most

00:02:05.300 published research in the biomedical field is incorrect, which is obviously out of the gate,

00:02:10.440 a staggering statement. We go on to discuss a number of his other seminal papers,

00:02:14.800 and then really kind of tackle some of the hard issues in medical research, including

00:02:18.740 my favorite topic, nutritional epidemiology. As always, John is candid and full of insight,

00:02:25.760 so I'm just going to leave it at that and hope that you trust me and make time to listen to this

00:02:30.720 one. So please, without further delay, enjoy my discussion with John Ioannidis.

00:02:35.860 John, this is really exciting for me to be as close to sitting down with you as I can be during this

00:02:46.460 time. I've been wanting to interview you for as long as I've had a podcast, and obviously,

00:02:51.280 we've known each other for probably close to 10 years now. Of course, you first came on my radar

00:02:56.820 in 2005 with a paper that we're going to spend a lot of time discussing today. But

00:03:02.000 before we get to that, how would you describe yourself to people? Because you have such a

00:03:07.760 unique background. I think that it's very difficult to know yourself, and I've been struggling

00:03:14.340 on that front for a long time. So I'm trying to be a scientist. I think that this is not an easy job.

00:03:21.840 It means that you need to reinvent yourself all the time. You need to search for new frontiers,

00:03:28.600 for new questions, for new ways to correct errors and to correct your previous self

00:03:35.140 in some way. So under that denominator of scientists in the works, probably it would be a good place to

00:03:45.880 put my whereabouts. Now, your background is also in mathematics. And I think that's part of my

00:03:52.160 appreciation for you is the rigor with which you bring mathematics to the study of science. And in

00:03:57.680 particular, we're going to discuss some of your work and how you use mathematical models as tools

00:04:03.020 to create frameworks around this. Now, you were born in the US, but grew up in Greece. Is that correct?

00:04:08.940 Indeed. I was born in New York, in New York City, but I grew up in Athens. And I always loved mathematics.

00:04:16.020 I think that mathematics are the foundation of so many things. And they can really transform our

00:04:22.400 approach to questions that without mathematics, it would be very difficult to make much progress.

00:04:28.740 How did you navigate your studies? Because you were obviously very prolific in mathematics. If I

00:04:34.740 recall reading somewhere in one of your bios, you even won the highest honor that a graduating college

00:04:41.200 student could win in mathematics in Greece at the time. How did you decide to also pursue something in

00:04:48.100 the biological sciences in parallel as opposed to staying purely in the natural or philosophical

00:04:53.360 sciences of mathematics? Medicine had the attraction of being a profession where you can save lives.

00:05:00.060 And I think that intellectual curiosity is very interesting, but the ability to make a difference

00:05:04.920 for human beings and to save lives, to improve their quality of life, seemed to be, at least in my eyes

00:05:12.560 as a young person, something that was worthwhile pursuing. I had a very hard time to choose what

00:05:19.860 pieces of mathematics and science and medicine I could combine in what I wanted to do. I think that I

00:05:27.940 have tried my hands in very different things. I have probably failed in all of them. But in some ways,

00:05:33.400 I saw that these were complementary. So I believe that medicine is amazing in terms of its possibilities

00:05:39.880 to help people. You need, however, very rigorous science. You need very rigorous scientific method

00:05:46.320 to be applied if you want to get reliable evidence. Then you also need quantitative approaches. You need

00:05:52.260 quantitative tools to be able to do that. So I think that none of them is possible to dispense without

00:05:58.620 really losing the hole and losing the opportunity to do something that really matters eventually.

00:06:05.360 And your parents were physicians as well. Is that correct?

00:06:07.920 Indeed. Both of them were physicians, actually physician scientists. So I did have an early

00:06:14.440 exposure to an environment where I could hear their stories of clinical exposure. At the same time,

00:06:22.440 I could see them working on their research. I remember these big tables with scientific papers spread

00:06:28.580 all over them and with what were the early versions of computerized research. I think that I had the

00:06:36.360 chance to be exposed to software and computers in an early phase because my father and my parents were

00:06:43.420 interested in doing research. So you finished medical school and your postgraduate training also in

00:06:49.280 Greece, or did you do part of that in the United States? I finished medical school in Greece in Athens,

00:06:54.760 in the National University of Athens. And then I went to Harvard for residency training. And then Tufts

00:07:00.740 General Medical Center for training in infectious diseases. At the same time, I was also doing joint

00:07:06.880 training in healthcare research. So it was very interesting and fascinating years learning from

00:07:14.140 great people.

00:07:15.840 And who were some of the people that you think back as having kind of shaped your thinking during those

00:07:21.900 years?

00:07:22.140 In the medical school, I had some great teachers. One of them was the professor of epidemiology,

00:07:28.240 Dmitry Trikopoulos, who was also chair of epidemiology at Harvard. And he had some really great

00:07:34.500 statisticians in his team. So from the first year at medical school, I went to meet them and tried to use

00:07:42.240 every textbook that they could give me and every resource that I could play with. In my residency

00:07:49.280 training, I was very fortunate to meet a great physician scientist, especially in infectious

00:07:55.100 diseases, actually. Bob Mellering was the physician in chief and professor at Harvard, professor of

00:08:02.000 medical research as well. And he was really an amazing personality in terms of his clinical acumen and

00:08:10.240 his approach to patients. Also, his very temperate mode of dealing with very serious problems and

00:08:17.520 dissecting through the evidence in trying to make decisions and, of course, start with making

00:08:22.840 diagnosis. At the end of my residency training, I had the pleasure to meet the late Tom Chalmers,

00:08:33.080 along with Joe Lau. They were at Tufts at that time. And my meeting with them was really a revelation

00:08:39.860 because they were the ones who were advancing the frontiers of evidence-based medicine. Evidence-based

00:08:45.800 medicine had just been coined as a term, pretty much, by the McMaster team, David Sackett and Gordon

00:08:52.020 Wyatt. And Tom Chalmers was the first person in the U.S. to design a randomized trial. He was also one of

00:09:00.880 the first to perform meta-analysis that had a major impact in medical science. At the time that I met them,

00:09:08.080 they had just published an influential paper on cumulative meta-analysis in the New England Journal of

00:09:12.680 Medicine. And it was a revelation for me because somehow what they were proposing was mixing

00:09:19.240 mathematics, rigorous methods, evidence, and medicine in one coherent whole, which seemed to be

00:09:28.300 a forlorn hope until then for me. I was just seeing lots of clinical exposures that there was very little

00:09:35.180 evidence to guide us. There was no data or very poor data and a lot of expert-based opinion guiding

00:09:41.460 everything that was being done. And so this is just temporarily, I mean, Chalmers died in the

00:09:46.760 mid-90s. So this is what, the early 90s that you were fortunate enough to meet him? Yes. I met him in

00:09:53.460 1992 and he died about five years later. I was grateful that I had the opportunity to work with him

00:10:00.460 and also with Joseph Lau, who was at that time at Tufts Union Medical Center, which I went eventually to

00:10:07.500 do my fellowship training. Because there are so many things I want to talk about, John, and we don't

00:10:13.480 have the luxury of spending 12 hours together. I'm going to fast forward about a decade. I'm going to

00:10:19.020 fast forward to 2005, to that paper that I alluded to at the outset, which was the first time your work

00:10:26.480 came onto my radar, which is not to say anything other than that's just the first time I became aware

00:10:33.280 of sort of the gravity of your thinking. Can you talk a little bit about that? It was in PLOS One,

00:10:39.900 was that paper, correct? Yes. It was in PLOS Medicine.

00:10:43.860 PLOS Medicine. Okay. So this is basically an open source journal that I think another Stanford

00:10:49.600 professor actually was one of the guys behind this journal, if I recall. Pat Brown was one of the

00:10:54.540 forces behind PLOS, correct? Well, it was a transformative move at that time,

00:11:00.620 trying to create a new standard for medical journals. I think that now this has become

00:11:05.320 very widespread in a way. But I think back then it was something new, something that was a new

00:11:11.780 frontier in a sense. So you wrote a paper that on the surface seems, I mean, highly provocative,

00:11:21.040 right? The title of the paper is something to the effect of why most published clinical research

00:11:25.740 is untrue. I mean, that's the gist of it. Can you walk people through the methodology of this?

00:11:31.320 It's a theoretical paper, but explain to people who maybe don't have the understanding of mathematics

00:11:37.800 that you do, how you were able to come to such a stark conclusion, which I want to point out one

00:11:45.180 thing. I'll give you why I had an easy time believing the results of your paper is my mentor had shared

00:11:51.900 with me a statistic when I was, you know, sort of doing my postdoctoral training, which I found hard

00:11:58.580 to believe. But when I realized it was true, became the bookend to your claim. And that was some at the

00:12:04.980 time, something to the tune of 70% of published papers were never cited again, outside of auto

00:12:11.020 citation, meaning outside of the author citing his or her own work. And if you think about that for a

00:12:17.320 moment, if 70% of work can't even be cited by one additional person down the line, that tells you

00:12:25.180 it's, you know, either irrelevant or wrong. So again, that's not the same thing that you said,

00:12:30.560 but it at least primed me to kind of listen to the message you were talking about. So talk a little

00:12:36.820 bit about that paper. That paper, as you say, it's a mathematical model that is trying to match

00:12:42.840 empirical data that had accumulated over time, both in my work and also in the work of many other

00:12:49.000 scientists who were interested to understand the validity of different pieces of research that was

00:12:55.680 being produced. I think that many of us had been disillusioned that when evidence-based medicine

00:13:02.200 started, we thought that now we have some tool to be able to get very reliable evidence for decision

00:13:09.620 making. And very quickly, we realized that biases and results that could not be replicated and

00:13:15.820 results that were overturned and results that were unreliable were the vast majority. It was not

00:13:21.440 something uncommon. It was the rule that we had either unreliable evidence or actually, perhaps even

00:13:27.680 more commonly, no evidence. So it's an effort, that paper, to put a mathematical construct together

00:13:34.780 that would try to explain what is going on and would also try to predict in some ways what might

00:13:42.520 happen if some of the circumstances would change in terms of how we do research. So the model makes

00:13:49.840 for a framework that is trying to calculate what is the chance that if you come up with a Eureka,

00:13:58.000 a statistically significant result, that you claim I have found something, I have found some effect that is

00:14:03.520 not null. There is some treatment effect here, there is some not zero that I'm talking about. What are the

00:14:11.960 chances that this is indeed a non-null effect, that we're not seeing just a red herring? And in order to

00:14:18.760 calculate the chances that this is not just a red herring, you need to take into account what is your prior

00:14:26.380 chances that you might be finding something in the field that you're working. There are some fields that

00:14:31.960 probably have a higher chance of making discoveries compared to others. If you're unlucky to work in a

00:14:38.520 field that there's nothing to be discovered, you may be wasting your time and publishing one million

00:14:44.240 papers, but there's nothing to be discovered. So it's going to be one million papers that end up with

00:14:48.680 nothing. Conversely, there may be other fields that may be more rich in discovery, both the field and

00:14:55.700 the tools, the methods, the designs of the studies that we throw at trying to answer these questions

00:15:01.700 can be informative. The second component is in what environment of power are we operating? Meaning,

00:15:10.840 is the study large enough to be able to detect non-null effects of some size of interest? Or

00:15:18.260 maybe there are true effects out there, but our studies are very small, and therefore they're not

00:15:24.600 able to detect these effects. And in my experience until that time, I had seen again and again lots of

00:15:31.400 very small studies floating around with results that were very questionable, that could not be matched

00:15:37.320 with other efforts, especially when we were doing larger studies, most of them seemed to go away.

00:15:42.000 And power is important, not only because if you don't have enough power, you cannot detect things

00:15:50.120 that exist. What is equally bad or probably worse is that if you operate in an environment of low power,

00:15:57.440 when you do get something detected, it is likely to be false. And here comes the other factor that is

00:16:06.260 compounding the situation, bias, which means that you have some results that for whatever reason,

00:16:16.400 bias makes them to seem statistically significant, while they should not be. And bias could take

00:16:25.100 zillions of forms. I think that throughout my career, I feel like I'm struggling with bias, with my own

00:16:30.520 biases, and with biases that I see in the literature. But bias means that you could have conscious,

00:16:38.640 unconscious, or subconscious reasons why a result that should have been null somehow is transformed

00:16:44.740 into a significant signal. It could be publication bias, it could be selective reporting bias, it could be

00:16:52.820 multiple types of confounding bias, it could be information bias, it could be many, many other things that turn

00:16:59.420 null results into seemingly significant results while they are not. Then you have to take into account the

00:17:07.440 universe of the scientific workforce. We're not talking about a single scientist running all the studies.

00:17:15.300 It's not just a single scientist or a single team. We have currently about 35 million people who have

00:17:22.800 co-authored at least one scientific paper. We have many, many scientists who might be trying to attack

00:17:29.420 the same scientific question. And each one of them is contributing to that evidence. However,

00:17:35.880 there's an interplay of all these biases with all of these scientists. So if you take into account that

00:17:43.600 multi-scientist environment, multi-effort environment, you need to account for that in your calculations.

00:17:50.440 Because if you, for example, say, what are the chances that at least one of these scientists will find some

00:17:57.540 significant signal, this is a very different situation compared to just having one person

00:18:02.940 taking a shot and just taking a single shot. So this is pretty much what the model tried to take

00:18:08.920 into account, putting these factors together, and then trying to see what you get under realistic

00:18:16.700 circumstances for these factors. These factors would vary from one field to another. They would be

00:18:23.000 different, for example, if we're talking about exploratory research with observational data versus

00:18:29.920 small randomized trials versus very large phase three or even mega trials. It would be different if we're

00:18:37.700 talking about massive testing, like what we do in genetics versus highly focused testing of just one

00:18:44.860 highly specified pre-registered hypothesis that is being attacked. Running the calculations, the model

00:18:52.720 shows that in most circumstances where both biomedical research, but I would say most other fields of

00:19:01.360 research are operating, if you get a nominally statistically significant signal with a traditional

00:19:08.080 p-value of slightly less than 0.05, then the chances that you have a red herring, that this is not true,

00:19:16.720 that it is a false positive, are higher than 50%. There's a huge gradient, and in some cases it may be

00:19:25.920 much lower. The false positive rate may be much, much lower, and in others it would be much higher.

00:19:32.480 But in most circumstances, the chances that you got it wrong are pretty high. They're very high.

00:19:39.580 That's actually a very elegant description of that paper. I want to go back and unpack a few things

00:19:43.800 for people who maybe don't have some of the acumen down. So let's go a bit deeper into what a p-value

00:19:50.520 is. Everybody hears about it, and everybody hears the term statistically significant. So maybe explain

00:19:59.240 what a p-value is, explain statistical significance, and explain why it's not necessarily the same as

00:20:06.520 clinical significance and why we shouldn't confuse them. I think that there's major misconceptions

00:20:11.480 around significance. What we care in medicine is clinical significance, meaning if I do something,

00:20:17.880 or if I don't do something, would that make a difference to my patient? Or it could be in public

00:20:22.440 health, to the community, to cohorts of people, to healthy people who want to have preventive measures,

00:20:28.360 and so forth. Do I make a difference? Does it matter? Is it big enough that is worthwhile the cost,

00:20:35.560 the potential harms, the implementation effort, perhaps other alternatives that I have? How does

00:20:42.520 that compare to these alternatives? Maybe they're better, or cheaper, or easier to implement, or have

00:20:49.400 fewer harms? So this is really what we want to answer, but unfortunately most of the time we are stuck with

00:20:57.880 trying to answer a very plain frequentist approach question, which boils down to statistical significance.

00:21:06.600 Typically this boils down to a p-value threshold of 0.05 for most scientific fields. Over the years

00:21:13.960 there's many scientific fields that have diversified, and they have asked for more stringent levels of

00:21:19.400 statistical significance. A couple of years ago, along with many other people, we suggested that fields that

00:21:24.600 have not diversified and they do not adjust their levels of statistical significance to more stringency.

00:21:31.960 By default, they should be using a more stringent threshold. For example, use a threshold of 0.005 instead

00:21:38.680 of 0.05. However, most scientists are trained with statistics light to use some statistical test that

00:21:49.720 gives you some statistic that eventually translates to a p-value. And what that p-value means, it needs to

00:21:56.680 be interpreted as what are the chances that if I had an infinite number of studies like this one, I would

00:22:04.680 get a result that would be as extreme or more extreme. And even that is not a complete definition because

00:22:13.400 it does not take into account bias. Because maybe you would get a result that is as extreme, but it's

00:22:22.520 largely because of bias. For example, there's many, many fields that you can easily get p-values that are

00:22:29.720 astronomical. They're not just less than 0.005, but they may be 10 to the minus 100 with some of the large

00:22:38.120 databases that we have. We can easily get to astronomically small p-values. But this doesn't

00:22:45.000 mean much. It could just mean that you have bias and this is why you get all these astronomically low

00:22:50.360 p-values, but they don't really mean that the chance of getting such an extreme result is extremely

00:22:57.000 implausible and that there's something there. It just means that certainly there's bias. No more than that.

00:23:03.640 There has been what I call the statistics wars over the last several decades. People have tried to

00:23:09.480 diminish the emphasis on statistical significance. I think I have been in the camp of those who have

00:23:15.080 argued that we should diminish emphasis or at least try to improve the understanding of what that means

00:23:21.240 for people who use and interpret these p-values. In the last few years, this has become probably more

00:23:28.280 aggressive. Many great methodologists have suggested that we should completely abandon statistical

00:23:34.440 significance, that we should just ban the term, never use it again, and just focus on effect sizes,

00:23:41.960 focus on how much uncertainty we have about effect sizes, focus on perhaps Bayesian interpretation of

00:23:48.200 research. I have been a little bit reluctant about adopting the ban statistical significance approach

00:23:56.680 because I'm afraid that we have all these millions of scientists who are probably not very properly

00:24:05.080 trained to understand statistical significance, but they're completely not trained at all to understand

00:24:12.600 anything else that would replace it. So, in some ways, for some types of designs also, I would argue

00:24:19.560 that if you pre-specify and if you are very careful in registering your hypotheses and you have a protocol

00:24:26.040 that you deposit, for example, what is happening or should be happening with randomized trials, and you have

00:24:32.040 worked through this, that it makes sense that your hypothesis is clinically important, that the effect

00:24:38.680 size that you're trying to pick is clinically meaningful, it is clinically significant, then I would argue

00:24:45.800 that statistical significance and using a p-value threshold, whatever that is, depending on how you

00:24:51.800 design the study, makes perfect sense. It's actually a very transparent way of having some rules of the game

00:25:00.920 that then you try to see whether you manage to succeed or not. So, if you remove

00:25:07.640 these rules of the game after the fact in these situations, it may make things worse because you

00:25:11.880 will have a situation where people will just get some results and then they will be completely open

00:25:17.480 to interpret them as they wish. And we see that they interpret them as they wish even now without

00:25:22.520 any rules in the game, or at least by removing those rules post-hog. But if we could have some rules

00:25:29.720 for some types of research, I think that this is useful. For other types of research, I'm willing to

00:25:34.680 promote better ways of interpreting results, but this is not going to happen overnight. We have to take

00:25:41.720 for granted that most scientists are not really well trained in statistics, and they will misuse and

00:25:48.120 misinterpret and misapply statistics, unfortunately. So, we need to find ways that we will minimize the harm,

00:25:56.680 the harm. We will minimize the error and maximize in medicine, the clinically significant pieces,

00:26:04.440 and in other sciences, the true components of the research enterprise.

00:26:08.360 Now, at the other side of that statistical field is power, right? So, we go from alpha to beta.

00:26:18.040 And you alluded to it earlier. I want to come back to it because you actually said something very

00:26:22.440 interesting. I think most people who dabble enough in the literature understand that if you underpower

00:26:31.320 a study, so if you have too few samples, too few subjects, whatever the case might be,

00:26:39.400 and you fail to reach statistical significance, it's not clear that you failed to reach statistical

00:26:47.000 significance because you should be rejecting the null hypothesis or because you didn't have a large

00:26:51.160 enough sample size. So, that's always the fear, right? The fear is that you get a false negative.

00:26:58.280 But you said something else that I thought was very interesting, if I heard you correctly, which was,

00:27:03.560 no, you actually run the risk of a false positive as well if you're underpowered. Can you say more

00:27:09.720 about that? Indeed. In an underpowered environment, you run the risk of having higher rates of false

00:27:17.960 positives if you take the performance of the field at large. You know, if you take hundreds and thousands

00:27:23.320 of studies that are done in an underpowered environment. Even if you manage to detect the

00:27:28.680 real signals, you know, signals that do exist, if these signals are detected in an underpowered

00:27:34.440 environment, their estimates will be exaggerated compared to what the true magnitude is. And in many

00:27:42.680 situations, both in medicine and in other sciences, it's not important so much to find whether there's

00:27:48.840 some signal at all, which is what an null hypothesis is trying to work around. But how big is the signal? I mean,

00:27:56.680 if a treatment has a minuscule benefit, then I wouldn't care about it. I wouldn't use it because the cost and

00:28:03.560 the harms and everything on the other side of the balance is not making it worth it. So, most scientific fields,

00:28:12.600 have been operating in underpowered environments. And there's many reasons for that. And it varies

00:28:18.040 a little bit from one field to another, but there's some common denominators. Number one, we have a very

00:28:23.720 large number of scientists. Scientists are competitive. There's very limited resources for science. It means

00:28:29.640 that each one of us can get a very thin slice of resources. We need to prove that we can get significant

00:28:35.880 results so as to continue to be funded and to be able to advance in our career. This means that

00:28:43.240 we are stuck in a situation where we need to promote seemingly statistically significant results,

00:28:48.920 even if they're not. We need to do very small studies with these limited resources and then do

00:28:54.440 even more small studies rather than aim to do a more definitive large study. There's even a disincentive

00:29:01.160 incentive towards refuting results that are not correct because that means that you feel that

00:29:07.080 you're back to square zero. You cannot make a claim for continuing your funding. All the incentives,

00:29:11.960 at least until recently, have been aligned towards performing small studies in very selectively

00:29:19.160 reported circumstances and with flexibility in the way that results are analyzed and presented. And

00:29:25.320 I think that this leads to very high rates of results that are either completely false positives or they

00:29:33.240 may be pointing to some real signal, but the estimate of the magnitude of the signal is grossly exaggerated.

00:29:41.320 In recent years, we have started seeing the opposite phenomenon as well. We start seeing some fields that

00:29:48.840 have overpowered studies. Instead of just having very small studies, in some fields we have big data.

00:29:55.560 which means that you can access records, medical records from electronic health records on millions of

00:30:01.880 people, or you may have genetic information that is highly granular and gives you tons of information.

00:30:10.760 And big data are creating an opposite problem. It means that you're overpowered and you can get

00:30:18.120 statistically significant results that have no clinical meaning, that have no meaning really.

00:30:23.960 And even with a tiny little bit of bias, you may get all these signals just because bias is there.

00:30:31.160 So you're just measuring bias. You're just getting a big scale assessment of the distribution of bias in

00:30:38.840 your data sets. That's becoming more of a problem in some specific fields. I think that the growth of this

00:30:46.520 type of problem will be faster compared to the growth of the problem of small underpowered studies.

00:30:52.840 I think in most fields, it's a more common problem though, until now, that we have very small studies

00:30:58.840 rather than very large studies. Now, you've commented on GWAS studies. Do you want to talk a little bit

00:31:05.400 about that here? It sort of fits into this a little bit, doesn't it?

00:31:08.040 Genetics was something that I was very interested in from my early years of doing research because

00:31:15.560 it was a new frontier for quantitative approaches. Lots of very interesting methodology was being

00:31:20.920 developed in genetics. Many of the questions of evidence that had been stagnating in other

00:31:26.600 biomedical fields, they had a new opportunity to give us some new insights with much larger scale

00:31:32.680 evidence in genetics compared to what we had in the past when we were trying to measure things one at

00:31:38.760 a time, especially genetics was a fire hose of evidence in some way. So I found it very exciting.

00:31:46.120 And for many years, I did a lot of genetic research. I still do some. And very early on, we realized

00:31:53.640 through genetics that the approach that we had been following in most traditional epidemiology,

00:32:00.120 like looking at one risk factor at a time and trying to see whether it is associated with some

00:32:05.000 disease outcome was not getting very far. We could see in genetic epidemiology of candida genes

00:32:13.320 that most of these papers that were looking at one or a few genes at a time with association with

00:32:18.280 some outcome, just trying to cross the threshold of statistical significance and then claiming success,

00:32:23.720 they were just false positives. We saw that pretty early. It took some time for people to be convinced,

00:32:29.240 but then they were convinced and genetics took some steps to remedy this. They decided to do very

00:32:35.640 large studies to start with. They also decided to look at the entire genome, look at all the factors

00:32:41.800 rather than one at a time. And they also decided to join forces, not have each scientist try to publish

00:32:48.680 their results alone, but share everything, have a common protocol, put all the data together to maximize

00:32:55.880 power, to maximize standardization, to maximize transparency also, and then report the cumulative

00:33:04.120 results from the combined data from all the teams that had contributed to these large meta-analysis of

00:33:10.920 primary data. So this is a recipe that I think should be followed by many other fields, especially fields

00:33:18.840 that work with observational data in epidemiology. And some fields have started moving in that direction

00:33:24.280 as well, but not necessarily as much as the revolution that happened in genetics and population genomics.

00:33:31.320 So I was going to actually ask you exactly that question. I was going to save it for a bit later,

00:33:35.400 but let's do it now. Why did the field of genetics basically have the ability to self-police

00:33:41.480 and undergo this cultural shift in a way that, let's just put every card on the table here,

00:33:46.360 nutritional epidemiology has not. I mean, nutritional epidemiology, which we're going to spend a lot of

00:33:51.240 time talking about, is the antithesis of that. And it continues to propagate subpar information,

00:33:58.120 which is probably the kindest thing I could say about it. So what is it culturally about these two

00:34:03.880 fields that has produced such stark contrasts in the response to a crisis?

00:34:10.280 There's multiple factors. One reason is that genetics managed to have better tools for measurement

00:34:17.640 compared to nutritional epidemiology. We managed to decode the human genome. So we developed platforms

00:34:24.360 that could measure the entire variability more or less in the human genome with pretty high accuracy.

00:34:31.480 If you have genotypic platforms that have less than 0.01% error rate, this means that you have very

00:34:37.480 accurate measurement. As opposed to nutrition where the traditional tools have been questionnaires or

00:34:44.120 survey tools that have very high biases, very high recall bias, very low accuracy, and they do not

00:34:52.600 really capture the diversity of nutritional factors with equal granularity as we can capture the genetics

00:35:01.000 in their totality of the human genome. The second reason was that I believe in genetics there were no

00:35:07.400 strong priors, no strong beliefs, no strong opinions, no strong experts who would fight with their lives for

00:35:15.880 one gene variant versus another. We had some, you know, I think that some of us probably might have

00:35:21.320 published about one gene and then we would fiercely defend it because obviously if you publish a paper,

00:35:28.680 you don't want to be proven wrong, I think it's very human. But it was nothing compared to the scale that you see

00:35:35.320 in nutrition research where you have a very strong expert opinion base, people who have created careers and

00:35:43.880 they feel very strongly that this type of diet is saving lives and it should have policy implications,

00:35:50.680 it should change the world, it should change our guidelines, it should change everything.

00:35:55.240 Many of these beliefs are interspersed with religious or cultural or, you know, non-scientific beliefs

00:36:03.640 in shaping what we think is good diet. And as you realize, none of that really exists for genetics,

00:36:10.200 you know, polymorphism RS 2492-14 is unlikely to be endorsed by any religious, cultural, political,

00:36:20.600 or dietary proponents. It's a very different beast and I think that you can be more neutral

00:36:27.400 with genetics research because of this objectivity as opposed to nutrition where there's a lot of

00:36:34.120 heavy beliefs interspersed. Methodologically also genetics advanced faster. Nutrition has been stuck mostly

00:36:42.760 in the era of using p-values of 0.05 thresholds and using those thresholds in mostly post hoc research,

00:36:51.240 research that is not registered, that is selectively presented. People are trained in a way that they

00:36:58.920 need to play with the data, they need to torture the data, they need to try to unearth interesting

00:37:05.080 associations. And in some cases, of course, this becomes extreme like what we have seen in the

00:37:11.000 Cornell case where, you know, pretty much goes into the situation where you have fraud. I mean,

00:37:17.400 it's not just poorly done research, it's fraudulent research. But fraudulent research aside, even

00:37:24.120 research that is not fraudulent in nutrition has some standards of methods that are pretty suboptimal

00:37:29.640 compared to what genetics has adopted, that they decided that we have such a huge multiplicity that

00:37:36.200 we need to account for that. So, you know, we're not going to claim success for a p-value of 0.05.

00:37:41.480 We will claim success for a p-value of 10 to the minus 8. And if it's not that low, then forget it.

00:37:47.640 That's not really a finding. We need to get more data before we can say whether we have a finding or not.

00:37:53.960 Or they decided that they will share data, that they will create large coalitions of researchers

00:37:59.000 who would all share their data. They would standardize their data. They will standardize

00:38:02.440 the analysis. They would perform analysis in a very specific way. And they would also sometimes,

00:38:09.880 actually, I think this is becoming the norm, have two or three analyst teams analyze the same data

00:38:15.480 and make sure that they get the same results. These principles and these practices have started

00:38:21.400 being used in fields like nutrition, but to a much lesser extent. And I think that gradually we will

00:38:27.000 see more of that, but it's going to take some time. So there's multiple scientific and behavioral and

00:38:35.640 cultural and statistical and methodological reasons why these fields have not progressed

00:38:41.720 at the same pace of revolutionizing their research practices.

00:38:46.120 Let's talk a little bit about Austin Bradford Hill. I'm guessing you didn't have a chance to meet him.

00:38:51.400 He died in 91. Would you have crossed paths with him at all?

00:38:54.520 No, I didn't have that fortune, unfortunately. Do you think he would be rolling around in his

00:39:01.080 grave right now if he saw what was being employed based on the criteria he set forth, which I also

00:39:08.200 want to talk about your thoughts around the revision of these. But even if you just take his 10 criteria,

00:39:13.000 which we'll go through for a moment as a bit of a background on epidemiology, do you think that

00:39:17.720 what he had in mind is what we're doing today? I think that Austin Bradford Hill was very thoughtful.

00:39:25.480 He was one of the fathers of epidemiology. And of course, he didn't have the measurement tools and

00:39:32.200 the capacity to run research in such large scale as we do today. But he was spot on in coming up with

00:39:40.040 good questions and asking the right questions, asking the important questions. So his criteria,

00:39:46.280 I don't think that he thought of them as criteria. And I don't think that he ever believed that they

00:39:52.600 should be applied as a hard rule to arbitrate that we have found something that is causal versus

00:40:00.040 something that is not causal. If you read through the paper, it's a classic. It's very obvious that he

00:40:06.840 has a very temperate approach. He has a very cautious approach. Basically, he says none of these items

00:40:13.400 is really bulletproof. I can always come up with an example where it doesn't work.

00:40:19.480 And I think that this is really telling what a great scientist he was, because indeed in science,

00:40:25.080 there's hardly anything that is bulletproof. I don't know. The laws of gravity might be bulletproof,

00:40:30.840 but even those, as you realize, they're just only down to atomic levels. Yeah, exactly. You know,

00:40:36.360 in the theory of relativity, they would start failing. So he was very cautious. I think that

00:40:42.360 paper had tremendous impact. I think that we have not been very cautious in moving forward with many

00:40:49.000 of our observational associations and the claims that we have made about them. I don't want to give

00:40:54.920 a nihilistic perspective. And I don't want to give, let's say, a very negative perspective of

00:41:00.600 epidemiology because we run the risk of entering the other side where you will have some science

00:41:05.880 deniers saying, so you're not certain. And therefore we can have more air pollution. You know, we can

00:41:12.520 have more pesticides. We can have more. That's not clearly the case. I mean, we have very solid evidence

00:41:19.480 for many observational associations. There's not the slightest doubt that tobacco is killing right

00:41:26.040 and left. It's likely to kill 1 billion people over the last century. Let's go through tobacco as

00:41:32.600 the poster child for Bradford Hill's criteria. So I'm going to rattle off the quote unquote criteria

00:41:40.120 and just use tobacco as a way to explain it. So let's start with strength. How does the association

00:41:47.480 between tobacco and lung cancer fit in terms of causality vis-a-vis this criteria of strength?

00:41:55.000 It is huge. I mean, we do not see odds ratios of 10, 20, and 30, as we see with tobacco, with many

00:42:02.840 types of cancer and with other outcomes like cardiovascular disease. And I think that that

00:42:09.880 really stands out. And we see that again and again and again. We see very strong signal. We see signals

00:42:16.520 that are highly replicable. And that's the exception. In most of what we do nowadays in epidemiology,

00:42:24.680 we don't see odds ratios of 20. If I see an odds ratio of 20 in my calculations, I'm almost certain

00:42:31.240 that I have something wrong. I always go back and check and I find an error that I have done.

00:42:35.880 Yeah, you're probably off by a log if you're getting a 20 nowadays.

00:42:41.880 Probably two log. I think in genetics, we are dealing actually with odds ratios of 1.01 at the time.

00:42:51.880 So 1.01 may still be real. And of course, you know, then the question is...

00:42:57.320 Is it clinically relevant? Yeah.

00:42:58.660 It's unlikely to be clinically relevant, but how much certainty can you get even for its presence?

00:43:04.180 Yeah. So the strength is huge. You really essentially covered the next one, which is

00:43:07.700 consistency. If you look at all of the studies in the 1950s and the 1960s, they were all really

00:43:13.060 moving in the same direction. And that's whether you looked at physicians who were smokers,

00:43:17.220 non-physicians who were smokers, whichever series of data you looked at, you basically saw this 10x

00:43:23.940 multiplier in smoking. And I think on average, it worked out to be about 14x. There was about a 14

00:43:31.620 times higher chance. I mean, that's a staggering number. What about specificity? What does specificity

00:43:38.900 refer to here? I think that if you have such strength and such consistency, I would probably not

00:43:47.540 worry that much about the rest of the criteria. You know, I think that criteria like specificity or

00:43:55.940 like analogy, they're far more soft in terms of what they would convey. And also, we just don't know

00:44:05.620 the nature of nature in how it operates. Many phenomena may be very specific, but it doesn't

00:44:12.900 have to be so. We should not take it for granted that we should see perfect specificity or low

00:44:18.580 specificity. We see many situations where you have multi-dimensional situations of causality.

00:44:25.460 You have multiple factors affecting some outcome, or you have one factor affecting multiple outcomes.

00:44:32.500 The density of the webs of causality can be highly unpredictable. So I would not worry that much

00:44:39.700 about other criteria if you have some strength and consistency being so impressive in these cases.

00:44:50.100 Now, in most cases, we don't have that, right? We'll get an odds ratio of 1.14, which of course is a 14%

00:44:58.100 relative increase as opposed to 14x. So in those situations, when strength and consistency are out

00:45:05.300 the window, which is essentially true of everything in nutritional epidemiology, I can't really think

00:45:10.820 of examples in nutritional epi where you have strength and consistency.

00:45:15.060 Well, major deficiencies, I think, would belong to the category of very clear signals.

00:45:20.740 Major nutritional deficiencies, you know, if you have like...

00:45:23.380 Yes, yeah, yeah.

00:45:24.100 You know, berry, berry, for example.

00:45:25.620 Sure, sure, yeah. Or, you know, thiamine deficiency where you're out to lunch, yeah.

00:45:30.100 Yeah. But do you then look at... I mean, even biological gradient gets very difficult

00:45:36.020 with the tools of nutritional epi. Do you start to look at experiment? Plausibilities,

00:45:41.060 to me, has always struck me as a very dangerous one because, I don't know, it just seems a bit of

00:45:47.620 hand-waving. I mean, where do you then look?

00:45:50.260 I think the first question is whether you can get experimental evidence. To me, that's the priority.

00:45:57.940 And I realize that in some circumstances, when you know that you're dealing with

00:46:02.260 highly likely harmful factors, you cannot really have equipoise to do randomized trials. But for

00:46:10.020 most situations in nutrition, to take nutrition as the example that we have been discussing,

00:46:15.140 you can do randomized trials. And actually, we have done randomized trials. It's not that we're not

00:46:19.140 doing randomized trials. We have done many thousands of randomized trials. Most of them,

00:46:23.780 unfortunately, are pretty small and underpowered. And they suffer from all the problems that we

00:46:29.140 discussed earlier with underpowered studies that are selectively reported, with no pre-registration,

00:46:34.740 and with kind of haphazardly done analysis and reporting. I mean, they're not necessarily better

00:46:40.660 than observational data that suffer from the same problems. But we also have a substantial number of

00:46:47.700 very large randomized trials in nutrition. We have over 200 large randomized trials. Most of those

00:46:54.820 focus on specific nutrients or supplementations. Some are looking at diets like Mediterranean diet.

00:47:03.460 And with very few exceptions, they do not really show the benefits that were suspected or were proposed

00:47:10.020 in the observational data. There are exceptions, but there are not that many. That, to me, suggests that

00:47:16.660 most likely the interpretation that most of the observational signals are false positives or

00:47:21.620 or substantially exaggerated is likely to be true. We shouldn't be throwing out the baby with the

00:47:26.980 bathwater. There may be some that are worth pursuing and that may be true. And I think that this means 1.00

00:47:31.780 that we need to do more trials. The counter argument would be that, well, in a randomized trial,

00:47:37.380 especially a large one, especially with long-term follow-up, people will not adhere to what you

00:47:41.620 tell them to do with their diet or nutrient intake or supplementation. My response to this is that

00:47:49.220 when it comes to getting evidence about what people should eat, that lack of adherence is part of the

00:47:56.740 game. It's part of real life. So if a specific diet is in theory better than another, but people cannot

00:48:04.580 adhere to that. It's not really better because people cannot use it. So I get the answer to the

00:48:09.460 question that I'm interested in, which is, is that something that will make a difference?

00:48:14.900 Of course, it does not prove that biochemically or in a perfect system or in the perfect human

00:48:21.540 who is eating like a robot, that would not be helpful. But I don't care about treating robots. I

00:48:28.340 care about managing and helping real people. I agree with that completely, John. I would throw

00:48:33.940 in one wrench to that, which is in a world of so much ambiguity and misinformation, I do think it's

00:48:41.380 important to separate efficacy from effectiveness. What you're of course saying is in the real world,

00:48:46.660 only effectiveness matters. So real world scenarios with real world people. But I still think there is

00:48:52.740 a time and a place for efficacy. We do have to know what is the optimal treatment under perfect

00:48:58.980 circumstances if we want to have any chance at, for example, informing policy. I'll give you an example.

00:49:06.740 Food stamps. Should food stamps preferentially target the use of certain foods over others? Well,

00:49:14.020 again, if you had really efficacious data saying this type of food is worse than that type of food,

00:49:20.820 you could steer people towards healthier foods. It could impact the way we subsidize certain foods.

00:49:26.740 In other words, it's really all about changing the food environment. So it's, it is very hard to follow.

00:49:32.420 I think any diet is that is not the standard American diet. So anytime you opt out of the

00:49:37.860 standard American diet, whether it be into a Mediterranean diet or a vegetarian diet or a 0.85

00:49:42.740 low carbohydrate diet, or basically anything that's not the crap that we're surrounded by requires an 0.90

00:49:48.900 enormous effort. And I think a big part of that is because there is still so much ambiguity around

00:49:56.340 what the optimal nutritional strategies are. We haven't answered the efficacy question because I

00:50:01.220 think we keep trying to answer the effectiveness question. I agree. And I think I would not abandon

00:50:06.980 efforts to get some insights on efficacy, but we're not really getting these insights the way that we have

00:50:13.540 been doing things. I think that if you want to get answers on efficacy, there are options. One is

00:50:20.260 through the experimental approach. So you can still run randomized trials, but you can do them under

00:50:25.940 very controlled supervised circumstances that, you know, people are in a physiology or metabolism

00:50:33.060 clinic that they're being followed very stringently on what they eat and what happens to them. And you can

00:50:37.780 measure very carefully these biochemical and physiological responses. I think that a second

00:50:43.220 approach in the observational world or between the observational and the randomized is Mendelian

00:50:49.060 randomization studies with the advent of genetics. We have lots of genetic instruments that may be used to

00:50:56.660 create designs that are fairly equivalent to a randomized design. So you can get some estimates that are not

00:51:04.500 perfect because Mendelian randomization has its own assumptions and sometimes these are violated, but at

00:51:10.180 least I think that they go a step forward in terms of the credibility of the signals that you get. And

00:51:18.420 then you have the pure observational evidence, which I don't want us to discard it completely. I think that

00:51:24.180 these are data we should need to use them. We just need to interpret them very cautiously. If we use some of the

00:51:29.700 machinery that we have learned to deploy in other fields, for example,

00:51:35.380 one approach is what I call the environment-wide or exposure-wide association testing. Instead of

00:51:40.260 testing and reporting on one nutrient at a time, you just run an analysis of

00:51:44.900 all the nutrients that you have collected information on

00:51:47.860 and you can also do it for all the outcomes that you have collected information on. So that would be an exposure

00:51:53.620 outcome-wide association study. And then you report the results, taking into account the multiplicity

00:52:00.020 and also the correlation structure between all these different exposures and outcomes.

00:52:04.420 You get a far more transparent and complete picture. And if you get signals that seem to be recurrent and

00:52:11.860 replicable across multiple data sets, multiple cohorts that you run these analysis, you start having

00:52:19.860 higher chances of these signals to be reflecting some reality. Still, it's not going to be perfect

00:52:27.300 because of all the problems that we mentioned, but it is better compared to what we do now where we just

00:52:32.500 go after finding yet one more association, one at a time, and coming up with yet another paper that is

00:52:39.060 likely to be very low credibility. John, if your 2005 paper on the frequency with which we were going to come

00:52:47.460 across valid scientific publications is arguably the one that's... Is that your most cited paper?

00:52:54.980 No, it's not the most highly cited. It's received, I think, close to 10,000 citations, but for example,

00:53:01.300 the Prisma statement for meta-analysis has received far more.

00:53:05.220 Okay. Well, I was going to assume that the 2005 paper was the most cited, but I was going to say the

00:53:11.620 most entertaining is your 2012 paper, which is the systematic cookbook review. And again, this is

00:53:21.300 just one of those things where I remember the moment this paper came out and just the absolute belly

00:53:27.540 laughing that I had reading this. And frankly, the sadness I had reading this because it is a sarcastic

00:53:34.820 commentary in a way on a problem that I think plagues this entire field. So in this paper,

00:53:42.580 you basically, I don't know if it was randomly, but you selected basically 50 common ingredients

00:53:50.180 from a cookbook, right? Was there any method behind how you did this or was it purely random?

00:53:55.620 Well, we used the Boston cookbook that has been published since the 19th century. And we randomly

00:54:05.780 chose ingredients by selecting pages and then within those, the recipes and the ingredients that

00:54:12.100 were in these recipes. So yes, it is 50 ingredients, a random choice thereof, and trying to map how many

00:54:19.860 of those have had published studies in the scientific literature in terms of their association with cancer

00:54:25.220 risk. And not surprisingly, almost all of them had some published studies associating them with

00:54:32.180 cancer risk. Even the exceptions were probably exceptions because of the way that we searched.

00:54:36.420 For example, we didn't find any study on vanilla, but there were studies on vanillin. So if we had changed

00:54:42.500 with, if we had screened with the names of the biochemical constituents of these ingredients, probably,

00:54:49.460 I guess all of them might have had some studies associating them with cancer risk.

00:54:53.060 How was this paper received by the nutritional epidemiology community?

00:54:58.500 I think it created lots of enemies and lots of friends. And I'm grateful for the enemies who some

00:55:05.460 of them have pushed back with constructive comments. I think that most people realize that we have a

00:55:12.100 problem. I think that even people who disagree with me on nutrition, I have great respect for them. And

00:55:18.820 I'm sure that they're well-intentioned. I think that at the bottom of their heart, it's not that they

00:55:23.940 want to do harm. They want to save lives. They want to improve nutrition. They want to improve our world.

00:55:29.140 So I think that it should be feasible to reach some synthesis of these different approaches and

00:55:36.580 these different trends. And I do see that even people who have used traditional methods do start using

00:55:43.780 some of the methods that we have proposed, for example, these exposure-wide approaches or trying

00:55:49.780 to come up with large consortia and meta-analysis of multiple cohorts to strengthen the results and

00:55:56.980 the standardization of the results. I worry a little bit about some of the transparency of these efforts.

00:56:04.660 To give you one example, I have always argued that if you can have large-scale meta-analysis of

00:56:14.100 multiple teams, ideally all the teams joining forces and publishing a common analysis with common

00:56:20.660 standards, and ideally these would be the best standards and the best statistical tools thrown at

00:56:25.940 the analysis, this is much better than having fragmented publications. So in some questions of

00:56:32.820 nutrition, I have seen that happen, but here's what goes wrong. The invitation goes to other

00:56:40.420 investigators who have already found results that square with the beliefs of the inviting investigator.

00:56:48.980 So there may be 3,000 teams out there and the invitation goes to the 100 teams that have claimed and

00:56:56.820 believe that there is that association. And then these data are cleaned, combined,

00:57:02.580 and analyzed in the way that has found the significant association already, and you have a conclusion

00:57:08.420 with an astronomically low p-value that here it is. We have concluded that our claim for a significant

00:57:14.660 association is indeed true, and here's a large meta-analysis. Now this is equally misleading or even

00:57:21.300 more misleading than the single studies, because you have cherry-picked studies based on what you already know

00:57:29.540 to be the case. And putting them together, you just magnify the cherry-picking, you just solidify the

00:57:36.180 cherry-picking. So one has to be very cautious. Magnitude and amount of evidence alone does not make things

00:57:45.940 better. Actually it can make things worse. You need to ask what is the foundational construct of how that

00:57:53.220 evidence has been generated and identified and synthesized. And in some cases it may be worse

00:58:00.900 than the single small studies that are fragmented, because some of them may not be affected by the

00:58:06.340 same biases. There also seem to be sort of institutional issues around this, right? I mean,

00:58:11.300 your alma mater has a very strong point of view on nutritional epidemiology, right?

00:58:16.420 I think this is unavoidable. There are schools of thought in any scientific field, and Harvard has

00:58:22.740 an amazing team of nutritional epidemiologists. I have great respect for them, even though probably

00:58:28.260 we do not agree on many issues. I think that we should look beyond, let's say, the personal

00:58:35.620 differences or opinion differences. I think that my opinion has less weight

00:58:43.060 than anyone else's weight in that regard. If I want to be true to my standards, I'm not trying

00:58:49.940 to promote something because it is an opinion. What I'm arguing is for better data, for better

00:58:56.500 evidence, for better synthesis, and more unbiased steps in generating the evidence, synthesizing the

00:59:04.500 evidence, and interpreting it. And I'm willing to see whatever result emerges by that process. I'm not

00:59:12.020 committed to any particular result. I would be extremely happy if we do these steps and we

00:59:18.900 come up with a conclusion that, oh, 99% of the nutritional associations that were proposed were

00:59:25.300 actually correct. I have absolutely no problem with that if we do it the right way. What I'm worried is

00:59:32.500 resistance to doing it the right way. I think your point earlier, though, about the difference between,

00:59:39.300 say, how the genetics community and the nutrition community were able to sort of approach this

00:59:45.300 problem, I don't think you can forget your second point, right? Which is, it's very difficult to overcome

00:59:51.300 prior beliefs. And when an individual has made an entire career of a set of beliefs, I think it requires a very

01:00:00.180 special person to be able to say, you know, that may have been incorrect. And that is independent of

01:00:07.380 what that belief is, by the way. That can be a belief that may be correct or may be fundamentally

01:00:14.020 incorrect. You know, it's funny. I recently saw this thing on Netflix. It was the kind of documentary

01:00:20.340 about this DB Cooper case. Do you know this DB Cooper case? It's the only unsolved act of US aviation

01:00:31.300 crime that's never been solved. So do you know this case, John, the guy who hijacked an airplane

01:00:35.940 and then jumped out the back in 1971? Oh, I may have heard of it somewhere,

01:00:40.660 but yeah, I don't recall it very well. Well, it's interesting in that this guy hijacks an airplane

01:00:46.020 with a bomb and requests that the plane be landed while they pick up $200,000 and four parachutes.

01:00:52.100 He then gets the plane to take back off and jumps out the back with the money. And he's never been

01:00:56.740 found. Nine years later, they found a little bit of the money. That's the only real clue. And this

01:01:01.940 documentary focused on four suspects, four of many suspects. And you basically hear the story of each of

01:01:10.020 the four suspects and each of the people who today are making the case for why it was their uncle

01:01:16.500 or their husband or whatever. And my wife and I are watching this and we're thinking it's interesting.

01:01:22.260 And at the end, I just said to her, I said, you know, this is a great

01:01:27.220 sort of example of human nature, which is I believe every one of those people truly believes that it was

01:01:34.820 their relative or friend or whomever who was D.B. Cooper. And yet I think all of them are wrong.

01:01:41.700 I think each of those four suspects is categorically not the person. And yet each of them, I am convinced

01:01:50.900 by their sincerity. And I think that's the problem is I don't think science should be able to be that

01:01:57.940 way. That's the problem I think I have with epidemiology is that I guess I'm just not convinced.

01:02:04.340 It's a science in the way that we talk about science. Well, we have to be cautious because

01:02:10.180 we are human and scientists have beliefs. And I think that there's nothing wrong with having beliefs.

01:02:15.940 I think the issue is, can we map these beliefs? Can we be transparent? Can we be as much restrained

01:02:24.260 about how these beliefs are influencing the contact of our research and the way that we interpret our

01:02:30.340 findings? It will never be perfect. We're not perfect. And I think that aiming to be perfect is

01:02:36.580 not tenable. But at a minimum, we should try to impose as many safeguards in the process as to

01:02:43.380 minimize the chances that we will fool ourselves. You know, not fool others, but fool ourselves to

01:02:48.260 start with us as Feynman would say. This is not easy in fields that have a very deeply entrenched

01:02:55.780 belief system. And I think nutrition is one such. Again, there's no bad intention here. People are

01:03:02.180 well-intentioned. They want to do good. I will open a parenthesis. Of course, there is some bad

01:03:07.060 intentions. There's big food. There's industry who wants to promote their products and sell whatever

01:03:12.660 they produce. And that's a different story. And it is another huge confounder, both in nutrition

01:03:20.420 and in other fields, that we have very high penetrance of financial conflicts. But I think

01:03:27.060 that non-financial conflicts can also be important. And at a minimum, we should try to be transparent

01:03:32.340 about them, try to communicate both to the external world, but also to our own selves,

01:03:39.300 what might be our non-financial conflicts and beliefs in starting to go down a specific path

01:03:47.300 of investigation and a specific interpretation of results. You referred to it very, very briefly

01:03:53.380 earlier. What were the exact details of the case of Brian Wansick at Cornell? That was a lot to do,

01:03:59.620 and it seemed that that went one step further. That seemed like there was something quite deliberate going

01:04:05.220 on. Well, in that case, it was revealed based on the communication of that professor with his students

01:04:11.620 that practically he was urging them to cut corners and to torture the data until they would get some

01:04:19.700 nice-looking result. And practically, he was packaging nice-looking results as soon as they would

01:04:25.940 become available based on that data torturing process. So the data torturing was the central force

01:04:32.740 in generating these dozens of papers that were creating a lot of interest, and probably they

01:04:39.380 were very influential, many of them in terms of decision-making. But if you create results and

01:04:47.300 significance in that fashion, obviously the chances that these would be reproducible results is very,

01:04:53.700 very limited. Yeah. And of course, he was a very prominent person in the field. It makes you wonder,

01:05:00.740 how often is this going on with someone maybe less prominent, where they're part of that 35 million

01:05:07.380 people who are out there authoring the, what are we, about 100,000 papers a month make their way

01:05:13.380 onto PubMed? I mean, it's an avalanche, right? We have a huge production of scientific papers,

01:05:19.380 as you say. And if you look across all sciences, probably we're talking about easily five million

01:05:25.620 papers added every year. And the number is accelerating every single year. Of course,

01:05:31.460 very few of them are both valid and useful. And it's very difficult to sort through all that

01:05:39.460 mountain of published information. I think that research practices are substandard in most scientific

01:05:45.940 fields for most of the research being done. There's a number of surveys that have been conducted

01:05:51.940 asking whether fraud is happening and whether suboptimal research practices are being applied.

01:05:59.620 The results are different depending on whether you ask the person being interviewed on whether they

01:06:04.820 are doing this or whether people in their immediate environment are doing this. So fraud, I think,

01:06:10.660 is uncommon. I don't think that fraud is a common thing in science. It does happen now and then,

01:06:17.300 but I don't think that it is a major threat in terms of the frequency. It is a threat in terms of

01:06:23.140 the visibility that it gets and the damage that it gets to the reputation of science as an enterprise.

01:06:29.380 But it's not common. What is extremely common is questionable research practices or harmful research

01:06:35.620 practices, which means cutting corners in different ways. And depending on how exactly you define that,

01:06:41.940 the percentage of people who might be cutting corners at some point is extremely high. It may be

01:06:46.980 approaching even 100% if you define it very broadly and if you include situations where people are not

01:06:53.300 really cognizant about the damage that they do or the suboptimal character of the approach that they're

01:07:00.340 taking and how it subverts the results and or the conclusions of the study. Now, how do you deal with

01:07:07.780 that? Do you deal with that with putting people away to jail or making them lose their jobs or making them

01:07:13.860 pay $1 million fines? I don't think that that would work because you would probably need to fire the

01:07:20.580 vast majority of the scientific workforce and all of these are good people. They're not there because

01:07:25.300 they're frauds. But you need to work through training, through sensitizing the community, having a

01:07:32.740 grassroots movement about realizing what the problems are, how you can avoid these traps, and how you can use

01:07:40.900 better methods, how you can use better inference tools, and how you can enhance the credibility of

01:07:48.500 your field at large. Not only your own research, but the whole field needs to move to a higher level.

01:07:55.220 And I think that no scientific field is perfect. There are different stages of maturity at different

01:08:01.060 stages of engagement with better methods. And this is happening in a continuous basis. It's an evolution

01:08:08.740 process. So it's not a one time that we did one thing and then science is going to be clean and

01:08:15.860 perfect from now on. It is a continuous struggle. And every day you can do things better or you can do

01:08:21.140 things worse. Of those 35 million people who are out there publishing science today, how many of them

01:08:29.700 do you think are really fit to be principal investigators and be the ones that are making the decisions about where the

01:08:37.140 resources go, what the questions are that should be asked, and what the real and final interpretation

01:08:43.140 is? I mean, that has to be a relatively small fraction of that large number, right?

01:08:47.220 Well, 35 million is the number of author IDs in Scopus. And even that one is a biased estimate,

01:08:55.140 like any estimate. It could be that you have a much, much smaller number of people who are

01:09:01.540 what we call principal investigators. The vast majority of people who have authored at least

01:09:06.340 one scientific paper have just authored a single scientific paper and they have just been co-authors.

01:09:12.020 So they may be students or staff or supporting staff in larger enterprises and they never assume

01:09:19.620 the role of leading research or designing research or being the key players in doing research.

01:09:27.540 There's a much smaller core of people who I would call principal investigators. We're talking probably

01:09:33.620 at a global level, if you take all sciences into account, probably there are less than 1 million.

01:09:38.980 But still, this is a huge number, of course. Their level of training, their level of how familiar

01:09:44.500 they are with best methods, their beliefs and priors and biases, it's very difficult to fathom.

01:09:52.340 Some people argue that we need less research, that probably we should cut back and really be more

01:10:01.780 demanding in asking for credentials and for training and for methodological rigor for people to be able

01:10:09.700 to lead research teams. I'm a bit skeptical about any approach that is starting with a claim we need to

01:10:16.820 cut back on research because I think that research and science eventually is the best thing that has

01:10:23.380 happened to humans. Science is the best thing that has happened to humans. And I think that if we say

01:10:29.060 we need to cut back on research because research is suboptimal, we may end up in a situation where you

01:10:33.460 create an even worse environment, where you have even more limited resources and you still have all

01:10:38.500 these millions of people struggling to get these even more limited resources, which means that they

01:10:44.900 have even more incentives to cut corners, they have even more incentives to come up with strikings,

01:10:49.620 splashing results, and then you have an even more unreliable literature. So less is not necessarily the

01:10:59.140 solution. Actually, it may be problematic. Improved standards, improved circumstances

01:11:06.740 of doing research, an improved environment of doing research is probably what we should struggle for.

01:11:13.540 Creating the background where someone who's really a great scientist and knows what he or she is doing

01:11:22.580 will get support and will be allowed to thrive. Also, allowed to look at things that have a high

01:11:31.860 risk of failing. I think that if we continue incentivizing people to get significant results,

01:11:37.940 no matter how that is defined, we are incentivizing people to do the wrong thing. We should incentivize

01:11:43.700 them to try really interesting ideas and to have a high chance of failing. This is perfectly fine. I think

01:11:51.060 if you don't fail, you're not going to succeed. So we need to be very careful with interventions that

01:11:58.100 happen at a science-wide level or even discipline-wide level. We do not want to destroy science. We want to

01:12:04.900 improve science. And some of the solutions, they run the risk of doing harm sometimes.

01:12:11.860 Based on your comment about the sort of the risk appetite that belongs in science,

01:12:16.980 to me it suggests an important role for philanthropy because industry obviously has a very clear

01:12:24.580 risk appetite that is going to be driven by a financial return. By definition, everybody involved

01:12:30.580 in that is a fiduciary, whether it be to a private or public shareholder. And therefore, it's not the time

01:12:36.500 to take risk for the sake of discovery. Conversely, at the other end of that spectrum,

01:12:41.620 it might seem like the government in the pure public sector should be funding risk. But given the

01:12:50.100 legislative process by which that money is given out and the lack of scientific training that is in

01:12:58.420 the people who are ultimately decision makers for that money, it also seems like a suboptimal place

01:13:05.780 to generate risk. That seems to be the place where you actually want to demonstrate a continued

01:13:12.100 winning career, even if you're not advancing knowledge in the most insightful way. And so

01:13:18.420 what that leaves is an enormous gap for risk, which I think has to be filled with philanthropic work.

01:13:24.980 Do you agree with that?

01:13:25.620 I agree that philanthropy is very important. No strings attached philanthropy can really be

01:13:32.020 catalytic in generating science that would be very difficult to fund otherwise. Of course,

01:13:38.660 public funding is also essential. And I think that we should make our best to make a convincing case

01:13:44.340 that public funding should increase and not decrease. As I said, decreasing public funding makes things

01:13:49.700 far, far worse for many reasons. I think that we need to realign some of our priorities on what is

01:13:57.700 being funded with each one of these mechanisms. Currently, a lot of public funding is given to generate

01:14:04.740 translational products that are then exploited immediately by companies who make money out of them.

01:14:10.500 And conversely, the testing of these products is paid by the industry. I find that very problematic

01:14:20.180 because the industry is financing and controlling the studies, primarily randomized trials or other types

01:14:27.860 of evaluation research that are judging whether these products that they're making money of are going to be

01:14:35.140 be promoted, used, become blockbusters, and so forth, which inherently has a tremendous conflict.

01:14:43.300 I would argue that the industry should really pay more for the translational research for developing

01:14:48.820 products through the early phases. And then public funding should go to testing whether these products

01:14:56.340 are really worth it, whether they are beneficial, whether they have benefits, whether they have no harms or

01:15:01.460 very limited harms. That research needs to be done with unconflicted funding and unconflicted investigators,

01:15:08.740 ideally through public funds. Of course, philanthropy can also contribute to that.

01:15:13.380 Philanthropy, I think, can play a major role in allowing people to pursue high-risk ideas

01:15:19.620 and things that probably other funders would have a hard time to fund. I think that public funds should

01:15:26.820 also go to high-risk ideas. The public should be informed that science is a very high-risk enterprise.

01:15:32.980 If you try to create a narrative, and I think that this is the traditional narrative that money from

01:15:40.500 taxpayers are used only for paying research grants, that each one of them is delivering

01:15:48.020 some important deliverables. I think this is a false narrative. Most grants, if they really look at

01:15:54.180 interesting questions, they will deliver nothing. Or at least, you know, they will deliver that,

01:15:59.300 sorry, we tried, we spent so much time, we spent so much effort, but we didn't really find something

01:16:04.580 that is interesting. We'll try again. We did our best. We had the best tools. We had the best scientists.

01:16:10.660 We applied the best methods. But we didn't find the new laws of physics. We didn't find a new drug.

01:16:17.860 We didn't find a new diagnostic test. We found nothing. That should be a very valid conclusion.

01:16:23.460 If you do it the right way, with the right tools, with the right methods, with the best scientists

01:16:27.700 being involved, putting down legitimate effort, we should be able to say, we found nothing. But out of

01:16:35.140 one thousand grants, we have five that found something. And that's what makes the difference. It's not that

01:16:41.940 each one of them made a huge contribution. It is these five out of one thousand in some fields and

01:16:47.460 in other fields, obviously, maybe a higher yield that eventually transformed the world.

01:16:52.180 I mean, this seems like a bit of a communications problem because that's clearly the venture

01:16:56.420 capital model that seems to work very well, which is on any given fund, your fund is made back by one

01:17:03.780 company or one bet. It's not an average. It's a very asymmetric bet. And similarly, when you look at

01:17:10.660 other landmark public high-risk funding things, the Manhattan Project, the Space Project,

01:17:16.900 these were upsettingly high-risk projects. And yet I don't get the sense that the public wasn't

01:17:23.380 standing behind those. So it almost seems like there's a disconnect in the way scientists communicate

01:17:28.900 their work to the public versus the way NASA did. I mean, NASA was a PR machine. And obviously,

01:17:34.980 in the case of the Manhattan Project, I think you're in the duress of war. But we can't lose

01:17:40.500 sight of the fact that the scientific community was the one that stood up. The physicists of the

01:17:44.980 day are the ones that said to Roosevelt, like, this has to be done. I mean, Einstein took a stand.

01:17:51.220 So I don't know. I guess it all comes back to scientists need to lead a bit and lead to be

01:17:57.060 better communicators with the public, right? Science communication is a very difficult business.

01:18:02.580 And I think that especially in environments that are polarized, that have lots of conflicts,

01:18:09.380 inherent conflicts, lots of stakeholders in the community are trying to achieve the most for

01:18:15.860 themselves and for their own benefits. It can be very tricky. You know, scientists have a voice,

01:18:22.260 but that voice is often drowned in the middle of all the screams and Twitter and social media and media

01:18:28.260 and agendas and lobbies and everything. How do we strengthen that? I think that

01:18:35.780 there's two paths here. One is to use the same tricks as lobbies do. And the other is to stick to

01:18:41.380 our guns and behave as scientists. You know, we are scientists, we should behave as scientists. I cannot

01:18:47.700 prove that one is better than the other. I think that both myself and many others feel very uneasy

01:18:53.860 when we are told to really cross the borders of science and try to become communicators that are

01:19:01.780 lobbying even for science. It's not easy. You want to avoid exaggeration. You want to say that I don't

01:19:08.900 know. I'm doing research because I don't know. I'm an expert, but I don't know. And this is why I believe

01:19:15.140 that we need to know because these are questions that could make a difference for you. How do you tell

01:19:20.340 people that most likely I will fail, that most likely a hundred people like me will fail, but maybe one

01:19:26.020 will succeed? We need to keep our honesty. We need to make communication clear-cut. We need to also fight

01:19:34.740 against people who are not scientists and who are promising much more. And they would say that,

01:19:39.780 oh, you need to do this because it will be clearly a success. And they're not scientists,

01:19:44.260 but they're very good lobbies. It's very difficult. It's difficult times for science.

01:19:49.860 It's difficult times to defend science. I think that we need to defend our method. We need to defend

01:19:55.300 our principles. We need to defend the honesty of science in trying to communicate it rather than

01:20:02.180 build exaggerated promises or narratives that are not realistic. Then even if we do get the funds,

01:20:11.380 we have just told people lies. I completely agree. I don't think what you and I are saying

01:20:16.180 is mutually exclusive. I think that's the point, right? I mean, you said it a moment ago, right? I

01:20:20.420 mean, Feynman's famous line that, you know, the most important rule in science is not to fool anyone.

01:20:26.420 And that starts with yourself. You're the most, you're the easiest person to fool. And once you

01:20:30.900 fooled yourself, the game is over. And I think the humility that you talk about communicating with

01:20:36.260 the public is the necessary step. I think people, I mean, I guess for me, just having my daughter,

01:20:42.180 who's now just, you know, starting to understand, you know, or ask questions about science is so much

01:20:48.820 fun to be able to talk about this process of discovery and to remind ourselves that it's not

01:20:53.860 innate, right? This is not an innate skill. This is something, this methodology didn't exist 500 years

01:21:01.140 ago. So for all but 0.001% of our genetic lineage, we didn't even have this concept.

01:21:10.180 So that gives us a little bit of empathy for people who have no training, because if you weren't

01:21:16.420 trained in something, you know, it's, you know, there's no chance you're going to understand it

01:21:20.660 without this explanation. But I feel strongly that there can't be a vacuum, right? Because the vacuum

01:21:26.900 always gets filled. And if the scientists aren't the ones speaking, then, you know,

01:21:30.980 if the good scientists aren't the ones speaking, then it's either going to be the bad ones and or

01:21:34.980 the charlatans who will. Before we leave epi, there's one thing I want to go back to that I

01:21:40.420 think is another really interesting paper of yours. This is one from two years ago. This is the challenge

01:21:45.700 of reforming nutritional epidemiologic research. And this is the one where you looked at the single foods

01:21:53.940 foods and the claims that emerged in terms of epidemiology. I mean, some of these things were

01:22:01.160 simply absurd. Do you remember this paper that I'm talking about, John? You've written a couple

01:22:04.940 along these lines, but this is the one that, you know, where you found a publication that suggested

01:22:11.060 eating 12 hazelnuts per day extended life by 12 years, which was the same as drinking three cups of

01:22:19.180 coffee and eating one mandarin orange per day could extend lifespan by five years. Whereas consuming

01:22:26.700 one egg would shorten it by six years and two strips of bacon would shorten life by a decade, which by the

01:22:34.240 way, was more than smoking. How do you explain these results? And more importantly, what does it tell us

01:22:41.300 again about this process? Well, these estimates obviously are tongue in cheek. They're not real

01:22:47.860 estimates. They're a very crude translation of what the average person in the community would get if

01:22:53.700 they see the numbers that are reported, typically with relative risks in the communication of these

01:22:59.540 findings. They're not epidemiologically sound. You know, the true translation to change in life

01:23:06.420 expectancy would be much smaller. But even then, they would probably be too big compared to what the real

01:23:12.900 benefits might be or the real harms might be with these nutrients. I think it just shows the the magnitude

01:23:19.620 of the problem that if you have a system that is so complicated with so inaccurate measurements with so

01:23:24.420 convoluted and overtly correlated variables with selective reporting and biases superimposed, you get a

01:23:34.100 situation pretty much like what we described in the nutrients and cancer risk where

01:23:41.380 you get an implausible big picture where you're talking about huge effects that are unlikely to be true.

01:23:49.220 So it goes back to what we have been discussing about how you remedy that situation, how you bring

01:23:54.980 better methods and better training and better inferences to that land of irreproducible results.

01:24:02.580 Now in, gosh, it might have been 2013, 14, a very interesting study was published called PREDIMED,

01:24:12.180 which we'll spend a minute on. And it was interesting in that it was a clinical trial.

01:24:16.820 It had three arms and it relied on hard outcomes. Hard outcomes meaning mortality or morbidity of some

01:24:24.420 sort rather than just soft outcomes like a biomarker. If you had told me before the results came out,

01:24:33.540 this is the study, you're going to have a low fat arm and two Mediterranean arms that are going to be split

01:24:39.540 this way and this way. And we're going to be looking at primary prevention.

01:24:45.860 I would have said the likelihood you'll see a difference in these three groups is quite low because

01:24:51.460 it just didn't strike me as a very robust design.

01:24:54.900 But I guess to the author's credit, they had selected people that were sick enough that within,

01:25:00.180 you know, I think they had planned to go as long as seven or so years, but under five years,

01:25:04.500 they ended up stopping this study, given that the two arms in the Mediterranean arm,

01:25:09.860 one that was randomized to receive olive oil, the other, I believe, received nuts,

01:25:15.860 performed significantly better than the low fat arm. And that's sort of how the story went until

01:25:22.740 a couple of years later. What happened then?

01:25:25.220 So, here you have a situation where I have to disclose my own bias that I love the Mediterranean

01:25:33.380 diet and I have been a believer that this should be a great diet to use. I mean, I grew up in Athens and

01:25:39.700 obviously, it's something that I enjoy personally a lot. And I would be very happy to see huge benefits

01:25:46.660 with it. For many years, I was touting these results as, here you go, you have a large trial

01:25:52.260 that can show you big benefits on a clinical outcome. And actually, this is Mediterranean diet,

01:25:57.380 which is the diet that I prefer personally, even better.

01:26:00.580 And just to make the point, it was both statistically and clinically very significant.

01:26:06.580 Indeed. Beautiful result. Very nice looking. And I was very, very happy with that. I would use it as

01:26:12.260 an argument that here, here's how you can do it the right way and show clinically relevant results.

01:26:20.420 But then it was realized that, unfortunately, this trial was not really a randomized trial.

01:26:26.020 The randomization had been subverted, that a number of people had not actually been randomized

01:26:30.740 because of problems in the way that they were recruited. And therefore, the data were problematic.

01:26:36.420 You had a design where some of the trial was randomized and some of the trial was actually

01:26:40.980 observational. So, New England Journal of Medicine retracted and republished the study

01:26:46.980 with lots of additional analysis that tried to take care of that subversion of randomization

01:26:53.540 in different ways, excluding these people from the calculations and also using approaches to try to

01:27:01.780 correct for the imposed observational nature of some of the data.

01:27:05.460 The results did not change much, but it creates, of course, a very uneasy feeling that if really the

01:27:13.620 creme de la creme trial, the one that I adored and I admired, had such a major problem, you know, such a

01:27:21.060 major, basic, unbelievably simple problem in its very fundamental structure of how it was run,

01:27:29.700 how much trust can you put on other aspects of the trial that require even more sophistication

01:27:35.700 and even more care? For example, arbitration of outcomes or how you count outcomes. As you say,

01:27:43.140 this is a trial that originally was reported with limited follow-up compared to the original intention.

01:27:47.540 It was stopped at an interim analysis. The trial has had lengthier follow-up. It has published a

01:27:52.980 very large number of papers as secondary analysis, but still we lack what I would like to see as

01:28:01.380 a credible result. I mean, it's a tenuous, partly randomized trial and unfortunately doesn't have the

01:28:08.260 same credibility now compared to what I thought when it was a truly randomized trial and there was one

01:28:15.300 outcome that was reported and that seemed to be very nice. Now, it's a partly randomized, partly subverted

01:28:22.660 trial with, I don't know, 200-300 publications floating around with very different claims each time.

01:28:30.500 Most of them looking very nice but fragmented into that space of secondary analysis.

01:28:37.780 It doesn't mean that Mediterranean diet does not work. I mean, I still like to eat

01:28:42.340 things that fit to a Mediterranean diet and this is my bias, but it just gives one example of how

01:28:50.820 things can go wrong even when you have good intentions. I think that I can see that people

01:28:56.020 really wanted to do it wrong, but one has to be very cautious.

01:28:59.940 Yeah. I mean, I think for me, the takeaway, if I remember some of the details, which I might not,

01:29:05.860 I mean, one of the big issues was the randomization around the inner household subjects,

01:29:09.380 right? They wanted that you couldn't have people in the same house eating the different diets,

01:29:14.500 which is a totally reasonable thought. It just strikes me as sloppiness that it wasn't done

01:29:21.780 correctly in the first place. You know, the cost of doing a study, the cost and duration

01:29:27.620 of doing a study like that is so significant that it's just a shame that on the first go,

01:29:34.100 it's not, you know, it's not nailed because, you know, it could be seven years and a hundred million

01:29:40.020 dollars to do that again. This is true, but one has to take into account that in such an experiment,

01:29:45.860 you have a very large number of people who are involved and their level of methodological training

01:29:51.380 and their ability to understand what needs to be done may vary quite a bit. So it's very difficult

01:29:56.980 to secure that everyone involved in all the sites involved in the trial would do the right thing.

01:30:03.300 And I think that this is an issue also for other randomized trials that are multicenter.

01:30:08.820 Very often now we realize that because of the funding structure, since as we said,

01:30:13.780 there's very little funding from public agencies, most of the multicenter trials are done by the

01:30:19.220 industry. They try to impose some rigor and some standards, but they also have to recruit

01:30:25.860 patients from a very large number of sites, sometimes from countries and from teams

01:30:30.340 that have no expertise in clinical research. And then you can have situations where a lot of the data

01:30:37.540 may not necessarily be fraudulent, but they're collected by people who are not trained, who have

01:30:41.620 no expertise, who don't know what they're doing. And sometimes depending on the study design, especially

01:30:46.660 with unmasked trials or trials that lack allocation concealment or both, you can have severe bias

01:30:54.020 interfere even in studies that seemingly appear to be like the creme de la creme of large scale

01:31:00.980 experimental research. John, let's move on to one last topic, at least for now, which is the events of 2020.

01:31:11.140 In early April, I had this idea talking with someone on my team, which was, boy,

01:31:21.220 the seroprevalence of this thing might be far higher than the confirmed cases of this thing.

01:31:31.540 And if that were true, it would mean that the mortality from this virus is significantly lower

01:31:38.420 than what we believe. This was at a time when I think there was still a widespread belief that

01:31:44.580 five to 10% of people infected with this virus would be killed. And there were basically a nonstop

01:31:53.060 barrage of models suggesting two to 3 million Americans would die of this by the end of the year.

01:32:00.820 The first person I reached out to was David Allison. And I said, hey, David, what do you think

01:32:06.740 about doing an assessment of seropositivity in New York City? And he said, let's call John Ioannidis.

01:32:15.140 So we gave you a call that afternoon. It was a Saturday afternoon. We all hopped on a Zoom and

01:32:19.540 you said, well, guess what? I'm doing this right now in Santa Clara. And I don't think it had been

01:32:25.780 published yet, right? I mean, I think you had just basically got the data, right?

01:32:30.260 I believe that it was about that time. Yes.

01:32:32.980 Tell me a little bit about that study and what did it show? Because it was certainly one of the

01:32:36.660 first studies to suggest that basically the seropositivity was much higher than the confirmed

01:32:42.340 cases. This is a pair of two studies, actually. One was done in Santa Clara and the other was done in

01:32:48.180 LA County. And both of them, the design aimed to collect a substantial number of participants

01:32:55.140 and tried to see how many of them had antibodies to the virus, which means that they had been

01:33:01.140 infected perhaps at least a couple of weeks ago. And they were studies that Aaron Ben David and Jay

01:33:07.700 Bhattacharya led. And also we had colleagues from the University of Southern California also leading the

01:33:13.940 study in LA County. They were studies that I thought were very important to do. I was just one of many

01:33:21.380 co-investigators, but I feel very proud to have worked with that team. They were very devoted and

01:33:27.060 they really put together in the field an amazing amount of effort and very readily could get some

01:33:33.540 results that would be very useful to tell us more about how widely spread the virus is.

01:33:39.540 The results, I'm not sure whether you would call them surprising, shocking, anticipated. It depends on

01:33:45.380 what your prior would be. Personally, I was open to the possibilities of any result. I had no clue

01:33:51.860 how widely spread the virus would be. And this is why I thought these studies were so essential.

01:33:57.220 I had already published more than a month ago that by that time that we just don't know. We just don't

01:34:03.380 know whether we're talking about a disease that is very widely spread or very limited in its spread,

01:34:09.700 which also translates in an inverse mode to its infection fatality rate. If it's very widely

01:34:15.380 spread, the infection fatality rate per person is much lower. If it is very limited in its spread,

01:34:22.260 it means that fewer people are affected, but the infection fatality rate would be very high. So

01:34:27.620 whatever the answer would be, it would be an interesting answer. And the result was that

01:34:33.300 the virus was very widely spread, far more common compared to what we thought based on the number

01:34:40.260 of tests that we were doing and the number of PCR documented cases at that time. In the early months

01:34:46.340 of the pandemic, we were doing actually very few tests. So it's not surprising at all that the

01:34:51.700 under ascertainment would be huge. I think that once we started doing more tests and or in countries that

01:34:57.620 did more testing, the under ascertainment was different compared to places that were not

01:35:02.580 doing much testing or were doing close to no testing at all. I think that the result was

01:35:09.060 amazing. I felt that that was a very unique moment seeing these results when I first saw that that's

01:35:16.100 what we got, that it was about 50 times more common than we thought based on the documented cases,

01:35:22.580 but obviously generated a lot of attention and a lot of animosity because people had very strong priors.

01:35:27.780 I think it was very unfortunate that all that happened in a situation of a highly polarized,

01:35:33.780 toxic political environment. Somehow people were aligned with different political beliefs

01:35:40.740 as if a political belief should also be aligned with a scientific fact. It was just completely

01:35:48.500 horrible. So it created massive social media and media attention, both good and bad. And I think that

01:35:56.580 we were bombarded with comments both good and bad and criticism. I'm really grateful for the criticism

01:36:02.980 because obviously these were very delicate results that we had to be sure that we had the strongest

01:36:10.020 documentation for what we were saying. And we went through a number of iterations to try to address

01:36:16.740 these criticisms in the best possible way. In the long term, with several months down the road

01:36:22.900 hindsight, we see that these results are practically completely validated. We have now a very large

01:36:29.780 number of seroprevalence studies that have been done in very different places around the world. We see that

01:36:35.620 those studies that were done in early days had, as I said, the worst under ascertainment. We had tremendous

01:36:41.060 under ascertainment in several places around the world. Even in Santa Clara, there's another

01:36:46.180 data set that was included in the national survey of a study that was published in the Lancet about a

01:36:52.020 month ago on hemodialysis patients. And the infection rate, if you translated, that was a couple of

01:36:59.460 months after our study. If you translate it to an infection fatality rate, it's exactly identical to what

01:37:04.420 we had observed in early April. So the study has been validated. It has proven that the virus is a very

01:37:13.460 rapidly and very widely spreading virus, and you need to deal with it based on that profile. It is a virus

01:37:20.020 that can infect huge numbers of people. My estimate is as of early December, probably we may have close

01:37:28.660 to one billion people who have already been infected, you know, more or less around the world.

01:37:34.180 And there's a very steep risk gradient. There's lots of people who have practically no risk or

01:37:41.060 minimal risk of having a bad outcome. And there are some people who have tremendous risk of being

01:37:48.100 devastated. We have, for example, people in nursing homes who have 25% infection fatality rate. You know,

01:37:55.220 one out of four of these people, if they're infected, they will die. So it was one of the most

01:38:01.940 interesting experiences in my career, both of the fascination about seeing these results and also

01:38:08.980 the fascination and some of the intimidation of some of the reaction to these results in a very

01:38:19.620 toxic environment, unfortunately. I don't necessarily mean by name, but what forces were the most critical?

01:38:27.140 Presumably, these would be entities or individuals that wanted to continue to promote the idea that the

01:38:34.020 risk here warranted greater shutdown, slowdown. Help me understand a little bit more where some of the

01:38:40.580 vitriol came from. I think that there were many scientists who made useful comments. And as I said,

01:38:48.100 I'm very grateful for these comments because they helped improve the paper. And then there were

01:38:53.060 many people in social media. That includes some scientists who actually, however, were not epidemiologists.

01:38:59.140 Unfortunately, in the middle of this pandemic, we have seen lots of scientists who have no

01:39:03.860 relationship to epidemiology become kind of Twitter or Facebook epidemiologists all of a sudden and,

01:39:09.700 you know, have very vocal opinions about how things should be done. I remember a scientist who was

01:39:15.780 probably working in physics or not, who was sending emails every two hours to the principal

01:39:21.780 investigator. And I was CC'd in them saying, you have not corrected the paper yet.

01:39:29.460 And every two hours, you know, you have not correct the paper yet. I mean, his comment was wrong to start

01:39:34.420 with. But as we were working on revisions, as you realize, we did that with ultra speed responding

01:39:41.540 within record time to create a revised version and to post it. But even posting, it takes five days,

01:39:48.580 more or less. But what do you think was at the root of this anger directed towards you and the team?

01:39:56.260 Unfortunately, I think that the main reasons were not scientific. I think that most of the animosity

01:40:01.940 was related to the toxic political environment at the moment. And personally, I feel that it is

01:40:07.860 extremely important to to completely dissociate science from politics. Science should be free to

01:40:14.820 say what has been found with all the limitations and all the caveats. But, you know, be precise and

01:40:20.500 accurate. I would never want to think about what a politician is saying in a given time or given

01:40:28.420 circumstances and then modify my findings based on what one politician or another politician is saying.

01:40:33.700 So I think that one of the attacks that I received was that I have conservative ideology,

01:40:42.740 which is like the most stupendous claim that I can think of, you know, looking at my track record and

01:40:50.580 how much I have written about climate change and climate urgency and emergency and the problem with

01:40:57.620 with gun sales and actually, you know, gun sales becoming worse in the environment of the pandemic

01:41:03.140 and the need to promote science and the need to diminish injustice and the need to provide health,

01:41:09.140 good health to all people and to decrease poverty. You know, claiming me that I'm a supporter of

01:41:16.340 conservative ideology, sick conservative ideology is completely weird. And then smearing of all sorts

01:41:23.700 of words that the owner of an airline company had given $5,000 to Stanford, which I was not even aware

01:41:31.460 of. The funding of the trial, which I was not even the PI, was through a crowdsourcing mechanism going

01:41:37.700 to the Stanford Development Office, which I'd never heard of who were the people who had funded that.

01:41:43.060 And of course, none of that money came to me or to all the other investigators who completely volunteered

01:41:48.260 our time. We have received zero dollars for our research, but tons of smearing.

01:41:55.540 Sorry, just to clarify, John, you're saying the accusation was that because an airline had contributed

01:42:02.260 $5,000 to Stanford, for which you saw none of it, that your assessment was really a way to tell everybody

01:42:11.700 that the airlines should be back to flying. Yes. But, you know, I heard about it when the

01:42:18.100 BuzzFeed reporter told me about it. Yeah, yeah, of course. Yeah, yeah, of course. No, no, I get it.

01:42:21.860 I get it. So it's very weird. And, you know, because of all the attacks that we received, you know,

01:42:28.020 I received tons of emails that were hate mail and some of them threatening to me and my family. My mother,

01:42:35.860 she's 86 years old and there was a hoax circulated in social media that she had died of coronavirus and

01:42:43.380 her friends started calling at home to ask when the funeral would be. And when she heard that from

01:42:50.820 multiple friends, she had a life-threatening, hypertensive crisis. So these people really had a

01:42:56.980 very toxic response that did a lot of damage to me and to my family and to others as well. And I think

01:43:05.700 that it was very unfortunate. I asked Stanford to try to find out what was going on. And there was

01:43:12.900 a fact-finding process to try to realize, you know, why is that happening? And of course, it concluded

01:43:18.260 that there was absolutely no conflict of interest and nothing that had gone wrong in terms of any

01:43:23.460 potential conflict of interest. But this doesn't really solve the more major problem. For me, the most

01:43:31.300 major problem is how do we protect scientists? It's not about me. It is about other scientists,

01:43:37.300 some of them even more prominently attacked. I think one example is Tony Fauci. He was my supervisor.

01:43:43.620 I have tremendous respect for him. He was my supervisor when I was at NIH. He's a brilliant

01:43:49.380 scientist and he has been ferociously attacked. There's other scientists who are much younger.

01:43:55.220 They're not, let's say, as powerful. They would be very afraid to disseminate their scientific findings

01:44:01.860 objectively if they have to ponder what the environment is at the moment and what do different

01:44:08.100 politicians say and how will my results be seen. We need to protect those. We need to protect people

01:44:13.860 who would be very much afraid to talk and they would be silenced. If we see examples that, you know,

01:44:20.580 can you see what happened to John Ioannidis or what happened to Tony Fauci? If I were to say something,

01:44:26.820 I would be completely devastated. So I think that we need to be tolerant. We need to give science

01:44:35.460 an opportunity to do its job, to find useful information, to correct mistakes or improve on

01:44:44.740 methods. I mean, this is part of the scientific process, but not really throw all that smearing

01:44:51.700 and all that vicious vitriol to scientists. It's very dangerous, regardless of whether it comes from

01:44:59.140 people in one or another political party or one in another ideology. It ends up being the same. It ends

01:45:05.620 up being populist attacks of the worst possible sort, regardless of whether they come from the left or

01:45:11.780 right or middle or whatever part of the political spectrum. Well, I'm very sorry to hear that you had

01:45:17.380 to go through that, especially at the level of your family. I knew that you had been attacked a little

01:45:23.540 bit. I was not aware that it had spread to the extent that you described it. What do we do going

01:45:29.860 forward here? I mean, it still seems to be a largely partisan issue. There's a very clear left versus

01:45:38.180 right approach to this that seems mostly science agnostic. I think it's viewed as

01:45:49.860 unwise to have a changing opinion outside of science, right? I mean, in science,

01:45:55.140 that's a hallmark of a great thinker, right? Someone who can change their mind in the presence of new

01:46:00.180 information. That's a core competency of doing good science. In fact, much of what we've spoken about

01:46:06.580 today is the toxicity of not being able to update your priors and change your mind in the face of

01:46:12.100 new information. But yet somehow in politics, that is considered the biggest liability of all time.

01:46:18.260 Somehow in politics, anytime you change your mind, it's wishy-washy and you're weak,

01:46:25.060 and you don't know your ideology. There seems to be an incompatibility here. And in a crisis moment like

01:46:31.540 this, which is, this was a crisis, that seems to bring these things to the fore, right?

01:46:38.340 It is true. And I don't want to see that in a negative light necessarily, because somehow the

01:46:44.420 coronavirus crisis has brought science to the limelight in some positive ways. I think that

01:46:50.980 people do discuss more about science. It has become a topic of great interest. People see that their

01:46:57.220 lives depend on science. They feel that their world depends on science. What will happen

01:47:01.940 in the immediate future and mid-range future depends on science and how we interpret science

01:47:06.180 and how we use science. So in a way, suddenly we have had hundreds of millions, if not billions of

01:47:13.620 people become interested in science acutely. But obviously most of those, unfortunately,

01:47:19.940 given our horrible science education, they have no science education. And they use the tools of their

01:47:26.500 traditional society discourse, which is largely political and sectorized to try to deal with

01:47:34.020 scientific questions. And this is an explosive mix. I think it creates a great opportunity to communicate

01:47:41.940 more science and better science. At the same time, it makes science a hostage of all these lobbying

01:47:48.900 forces and all of this turmoil that is happening in the community.

01:47:52.260 Well, John, what are you most optimistic about? I mean, you have lots of time left in your career.

01:47:58.500 You're going to go on and do many more great things. You're going to be a provocateur.

01:48:04.100 What are you most excited and optimistic about in terms of the future of science and the type of work

01:48:10.900 that you're looking to advance? Well, I'm very excited to make sure that, and it does happen,

01:48:16.740 that there's so many things that I don't know. And every day I realize that there's even more things

01:48:21.780 that I don't know. I think that so far, if that continues happening and every day I can find out

01:48:28.660 about more things that I don't know, things that I thought were so, but actually they were wrong and

01:48:33.300 I need to correct them and find ways to correct them, then I really look forward to a good future

01:48:38.900 for science and a good future for humans. I think that we're just at the beginning. We're just at

01:48:44.260 the beginning of knowledge. And I feel like a little kid who just wants to learn a little bit more,

01:48:52.260 a little bit more each time. Well, John, the last time we were together in person,

01:48:56.660 we were in Palo Alto when we had a Mediterranean dinner. So I hope that sometime in 2021, that'll bring

01:49:02.820 us another chance for another flaky white fish and some lemon potatoes and whatever other yummy

01:49:09.300 things we had that evening. That would be wonderful. And I hope that it does increase

01:49:13.860 life expectancy as well. Although even if it doesn't, I think it's worth it. John, thanks so

01:49:20.020 much for your time today. Thank you, Peter. Thank you for listening to this week's episode of The Drive.

01:49:24.900 If you're interested in diving deeper into any topics we discuss, we've created a membership

01:49:29.300 program that allows us to bring you more in-depth exclusive content without relying on paid ads.

01:49:34.660 It's our goal to ensure members get back much more than the price of the subscription.

01:49:39.140 Now to that end, membership benefits include a bunch of things. One, totally kick-ass,

01:49:43.860 comprehensive podcast show notes that detail every topic, paper, person, thing we discuss on each

01:49:49.380 episode. The word on the street is nobody's show notes rival these. Monthly AMA episodes or

01:49:55.460 Ask Me Anything episodes, hearing these episodes completely. Access to our private podcast feed

01:50:00.820 that allows you to hear everything without having to listen to spiels like this. The Qualies,

01:50:06.180 which are a super short podcast that we release every Tuesday through Friday, highlighting the

01:50:10.820 best questions, topics, and tactics discussed on previous episodes of The Drive. This is a great way

01:50:15.700 to catch up on previous episodes without having to go back and necessarily listen to everyone.

01:50:20.580 Steep discounts on products that I believe in, but for which I'm not getting paid to endorse,

01:50:26.100 and a whole bunch of other benefits that we continue to trickle in as time goes on. If you

01:50:30.740 want to learn more and access these member-only benefits, you can head over to peteratiamd.com

01:50:35.380 forward slash subscribe. You can find me on Twitter, Instagram, and Facebook, all with the ID

01:50:42.020 peteratiamd. You can also leave us a review on Apple Podcasts or whatever podcast player you listen on.

01:50:48.740 This podcast is for general informational purposes only and does not constitute the practice of

01:50:53.780 medicine, nursing, or other professional healthcare services, including the giving

01:50:58.340 of medical advice. No doctor-patient relationship is formed. The use of this information and the

01:51:04.260 materials linked to this podcast is at the user's own risk. The content on this podcast is not intended

01:51:10.500 to be a substitute for professional medical advice, diagnosis, or treatment. Users should not disregard

01:51:17.460 or delay in obtaining medical advice from any medical condition they have, and they should seek

01:51:22.900 the assistance of their healthcare professionals for any such conditions. Finally, I take conflicts

01:51:28.660 of interest very seriously. For all of my disclosures and the companies I invest in or advise, please visit

01:51:35.140 peteratiamd.com forward slash about where I keep an up-to-date and active list of such companies.

01:52:05.140 Thank you.

01:52:12.340 Thank you.

01:52:14.340 Thank you.

01:52:16.340 Thank you.

The Peter Attia Drive - January 04, 2021

#143 - John Ioannidis, M.D., D.Sc.： Why most biomedical research is flawed, and how to improve it

Episode Stats

Summary

Transcript