Dr. Ben Goldacre and the Reproducibility of Research

The institution of peer-reviewed, published research is not a magic wand that ensures that all findings are disseminated to all relevant parties. I wrote in Genesis and Genes about an important paper that appeared in Nature in March 2012:

During a decade as head of global cancer research at Amgen, [Glenn] Begley identified 53 “landmark” publications – papers in top journals, from reputable labs – for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development. Result: 47 of the 53 studies (89%) could not be replicated. He described his findings in a commentary piece published in the journal Nature in March 2012. In a Reuters report, Begley said “It was shocking. These are the studies the pharmaceutical industry relies on to identify new targets for drug development… As we tried to reproduce these papers we became convinced you can’t take anything at face value.” Begley’s experience echoes a report from scientists at Bayer AG. In a 2011 paper titled Believe it or not, they analyzed in-house projects that built on “exciting published data” from basic science studies. “Often, key data could not be reproduced,” wrote Khusru Asadullah, vice president and head of target discovery at Bayer HealthCare in Berlin, and colleagues. Of 47 cancer projects at Bayer during 2011, less than one-quarter could reproduce previously reported findings, despite the efforts of three or four scientists working full time for up to a year. Bayer dropped the projects.

Bayer and Amgen found that the prestige of a journal was no guarantee a paper would be solid. “The scientific community assumes that the claims in a preclinical study can be taken at face value,” Begley and Lee Ellis of MD Anderson Cancer Center wrote in Nature. They and others fear the phenomenon is the product of a skewed system of incentives that has academics cutting corners to further their careers. Part way through his project to reproduce promising studies, Begley met for breakfast at a cancer conference with the lead scientist of one of the problematic studies. “We went through the paper line by line, figure by figure,” said Begley. “I explained that we re-did their experiment 50 times and never got their result. He said they’d done it six times and got this result once, but put it in the paper because it made the best story. It’s very disillusioning.”

A reader has now kindly drawn my attention to a TED talk by Dr. Ben Goldacre.[1] The talk is entitled What Doctors Don’t Know about the Drugs they Prescribe. [Goldacre refers to the Nature paper mentioned above at about 02:00]. The points made by Dr. Goldacre are relevant to all branches of science.

Early on in the talk, Dr. Goldacre refers to a case study of the drug Lorcainide. This drug was designed to suppress arrhythmia, irregular beating of the heart. It was thought that since patients often suffer from arrhythmia after a heart attack, suppressing irregular heart-beating may be salubrious. In a preliminary study, fifty heart-attack patients were given Lorcainide, while another fifty patients were given a placebo. In the first group, ten patients died; in the second, only one patient died. The researchers concluded that Lorcaininde was dangerous. But because this trial was deemed a failure – the drug had no commercial prospects – the study was never published. This had tragic consequences. Not knowing the results of the unpublished study, other research groups in the following years who also thought that arrhythmia-suppressing drugs have medical potential brought similar medicines to market. According to Goldacre, this led to the unnecessary death of more than 100 000 patients.

A little further in his talk Dr. Goldacre discusses the drug Reboxetin, manufactured by Pfizer. With some asperity, Dr. Goldacre announces that he was misled by the published results regarding this drug, as indeed he was. Seven clinical trials were conducted to test the effectiveness of Reboxetin. One trial produced a positive result i.e. the drug produced better results than a placebo, and was published. Six trials produced negative results – and were not published. Goldacre says that he – and presumably thousands of other doctors – freely prescribed this drug on the basis of the published results.

Next, Dr. Goldacre turns to all the antidepressant trials submitted to the FDA for approval over a 15-year period. There were 38 positive trials and 36 negative trials submitted to the FDA. But when one looks at the publication record of these studies in the peer-reviewed academic literature, one finds that of the 38 positive trials, 37 were published; of the 36 negative trials, only 3 were published.

Dr. Goldacre peppers his talk with pungent comments about the rotten state of things. He says that publication bias is so prevalent that it cuts to the core of evidence-based medicine; the state of affairs “is a systematic flaw in the core of medicine”; he uses phrases like “This is a disaster” and terms like “cancer” to describe the situation.

***

The issues discussed by Dr. Goldacre in his TED talks and in his books are by no means limited to medical science. Publication bias is a systemic problem within science. One manifestation of this is that research that does not conform to basic tenets of sundry paradigms is simply not published, thus distorting the impression that scientists and the public form about important topics.

I discussed this in some length in Genesis and Genes. I pointed out that the problems begin right at the outset of one’s university training. Laboratory exercises, in which students ostensibly investigate some phenomenon, are actually exercises in reproducing textbook results.[2] I quoted the distinguished evolutionary biologist Massimo Pigliucci (who has doctorates in genetics, botany, and the philosophy of science):

Things are often only marginally better in college or university classes… Worse yet, most of these exercises are “prepackaged” labs designed to obtain a predetermined outcome, which often enough does not occur because of the carelessness of both students and teaching assistants. The latter are then tempted to do the worst thing they could possibly do in teaching science: tell the students that they should have gotten result X instead, and to write up their reports as if they had. Is it a surprise, then, that the whole enterprise becomes meaningless and that most students think science is either too difficult for them to grasp or, worse, is actually done by cooking the results to come out according to a priori expectations…

I pointed out that in the vast majority of science programs, no modules are offered in the psychology of research or the history/philosophy of science. The result is that science undergraduates who are unaware of the myriad biases which affect research become scientists who are often unaware of the biases affecting the publication of research results.

In this context, I referred to a talk given by Professor Michael Merrifield, an astronomer at Nottingham University.[3] Merrifield was discussing a purely technical issue – measuring the distance between our Sun and the centre of the Milky Way galaxy. This is not anthropology or sociology, but the research is nonetheless subject to psychological forces:

And, more worrying, is something that scientists like to push under the carpet… there’s psychology in this as well. If, in 1985, I made a measurement of the distance [from the Sun] to the centre of the galaxy when everyone said it was ten kilo-parsecs, and I got an answer that said it was seven kilo-parsecs, I would have thought, “Well, I must have done something wrong” and I would have stuck it in some filing cabinet and forgot about it; whereas if I had got an answer that agreed with the consensus, I’d probably have published it… In this error process, there’s also psychology. As I say, scientists are very uncomfortable about this, because we have this idea that what we are doing is objective and above such things. But actually, there is a lot of human interaction and psychology in the way we do science.

This phenomenon – sticking your results into a filing cabinet because they stray uncomfortably far from the consensus – is common in contemporary science. The failure to publish anomalous or “wrong” results is as prevalent in astronomy, physics or biology as it is in medical science.

Publication bias is bad science (One of the books authored by Dr. Goldacre is entitled Bad Science, and one of his TED talks is called Battling Bad Science). And awareness of this aspect of modern science is crucial in the process of becoming an informed consumer of science. The picture conveyed to the public – whether the topic is climate change, evolutionary biology, cosmology or a host of other areas – is by no means one that reflects all the research being done. It is distorted by numerous factors. Sometimes, there are financial factors (as in the case of Lorcainide); sometimes, there are political factors (an example is climate science); sometimes, there are paradigm considerations (read about Daniel Shechtman, Robin Warren and Alfred Wegener). The bottom line is that the final picture is often significantly incomplete.

***

See also:

The post Replication of Experimental Data:

https://torahexplorer.com/2013/02/10/replication-of-experimental-data/.

References:

[1] See http://www.youtube.com/watch?v=RKmxL8VYy0M.

Retrieved 10th April 2013.

[2] A reader of Genesis and Genes who trained as an aeronautical engineer wrote to me to express how, upon reading this passage in the book, he was struck by the fact that this had happened to him repeatedly throughout his university career, without his being consciously aware of the phenomenon.

[3] See http://www.youtube.com/watch?v=gzvPH6A5CmQ

Retrieved 10th April 2013.

Advertisements

14 Responses to “Dr. Ben Goldacre and the Reproducibility of Research”

  1. Menachem Mevashir Says:

    Yoram,

    I’d appreciate if you could take a look at this article by David Pogge of Science Against Evolution: http://krsf.net/pdfs/TWWU120212.pdf

    Pogge is an adherent of the Seventh Day Adventist movement, as well as being a retired US Air Force engineer:
    http://scienceagainstevolution.info/pogge.htm
    http://krsf.net/archives.html

    He is very passionate about exposing what he believes are the falsified claims of evolutionists, as a quick glance at his website will demonstrate:
    http://scienceagainstevolution.info/index.shtml
    http://scienceagainstevolution.info/topics.htm

    My question to you is why you fight so hard for a literalist interpretation of Genesis? Pogge’s essay mentioned above makes abundantly clear that evolution is diametrically opposed to Christianity because of the twin beliefs that:

    (1) Death ensued in the world as a result of man’s fall or disobedience in the Garden of Eden, and

    (2) Jesus died and was resurrected in order to save us from the penalty of eternal death ensuing from this primal sin.

    Judaism does not espouse either of these two ideas (obviously), so why do find evolutionary claims so offensive. Many respectable rabbis and scientists like Moshe Tendler and Gerald Schroeder seem to have no problem seeing Genesis 1 as a kind of parable that is perfectly compatible with the claims of modern evolutionary science. You are the ONLY Orthodox Jewish scientist I am aware of who takes a literalist approach to the Genesis creation account.

    In the case of people like Pogge I can understand that they might have an underlying psychological need to justify their faith, which admittedly seems antithetical to Darwinist claims. But in your case you do NOT have such a hidden motivating factor.

    So this makes me wonder why you have staked out such an extreme and uncompromising position.

    Thanks for your candor.

  2. Matt Says:

    Hi Yoram,

    You seem to have the odd notion that scientists bumble around unaware of the psychological or sociological effects that could bias our results. Dealing with these sorts of issues is such an essential part of our job description that I can’t help but wonder what it is you think we do all day. The very *point* of the Scientific Method is the development of methodological protections against researcher bias. When we design a new experiment or embark on a new measurement, we discuss these potential pitfalls and set ground rules to protect against them. The problems are highly context dependent. In some cases we are biased against anomalous results. In other cases, incentives bias us towards anomalies that are not there. Each experiment is a special case.

    One of the most common and obvious solutions to these psychological biases is a technique known as “double-blinding”. This means that we rig our analysis code to prevent us from seeing the final result of our experiment. Only after preparing all of the analysis techniques, calibrating, and cross-checking on a separate data set do we finally “unblind” our measurement. In order to prevent ourselves from simply sticking anomalies “in the filing cabinet” we hand our analysis over to a third party. The unblinding procedure does not even happen until this review committee approves of our methods. The results are unblinded publicly, so there are no easy take-backs. If a result deviates strongly from all other past measurements, it boils down to a judgment call: publish or don’t publish? But, the call is not necessarily our own. It falls in the hands of our reviewers.

    Anomalies pop up regularly in my field. We don’t hide them. Many of these anomalies are strongly found to be artifacts of experimental error. But, I can provide you with countless examples of the researchers deciding to put their results out to the larger community anyway…even at the risk of humiliation if they are found to have messed up. For a recent example, look up the “superluminal neutrino” results from the Opera experiment. Whenever these sorts of anomalies pop up, they spark vigorous community discussions. But, the point is we are hyper-aware of these problems. And, we like to draw from experiences (good and bad) when we educate the public about how science works. This is the point of the wonderful video you linked to.

    **********
    “The failure to publish anomalous or “wrong” results is as prevalent in astronomy, physics or biology as it is in medical science.”
    ***********

    You never established how “prevalent” this effect is in any quantifiable way. More importantly, you provide no substantiation that it is “as prevalent” in physics or astronomy as it is In medical science. I can list hundreds of ways where astronomy and physics differ from medical research. Medical research is unrecognizably different: in collaboration size, speed of the publication cycle, sheer volume of published research, sample sizes, techniques, and technical challenges.

    Likewise you cannot draw broad conclusions about “a lack of reproducibility”. In my field (particle physics) all results are reproduced. We always have two competing experiments running by design. I challenge you to present an example of a significant measurement in particle physics or astrophysics that isn’t either reproduced multiple times or where efforts are not solidly underway to reproduce them.

    Your hand-picked medical science anecdotes cannot simply be generalized to all fields. Period.

    ***********
    “I pointed out that in the vast majority of science programs, no modules are offered in the psychology of research or the history/philosophy of science. The result is that science undergraduates who are unaware of the myriad biases which affect research become scientists who are often unaware of the biases affecting the publication of research results.”
    ***********
    Not all of education happens in the classroom or in one particular classroom. History is an integral part of physics education, for example (physics is unique in that it is typically taught in historic order rather than conceptual order). For PhD scientists, undergraduate education is largely irrelevant. It is at the graduate level where we are put to the test, and where we learn most of what we need to know. It is at this stage, under the mentorship of countless individuals and in the context of original research where most scientists hone their attention to all of these sorts of methodological issues. Again, it’s our job. It is what we care about and it’s what we spend a tremendous amount of energy on.
    You never answer my question on this point: How many “modules” have you taken on philosophy of science? What are your qualifications to play the role of “historian/philosopher of science”?

    It is all too easy to exaggerate the weaknesses and cast doubt upon the scholarship of others. It is much harder to scrutinize one’s own scholarship. What sort of care do you take to ensure the objectivity of your research?

  3. Searle Says:

    Dear Menachem,

    In brief response to your comment, and perhaps preempting what R’ Bogacz would say, I suggest you read Genesis and Genes (if you haven’t already done so). R’ Bogacz meticulously sites numerous Torah sources which support a literal understanding of Genesis 1.

    Besides, as a proof by counter-example, Rabbi Dr Gottlieb (also a scientist) espouses the same view regarding a literal interpretation as does R’ Bogacz 🙂

    Kol tov,
    Searle

  4. Menachem Mevashir Says:

    Right Searle. I know there are Torah sources that say this. My question is why? Those Torah sources were for the most part pre-evolutionary theory and pre-modern science. Most Jews are adaptable enough to find ways to make the Torah conform to contemporary knowledge. Why do Yoram and others not? And Gottlieb is no proof since he is no scientist, but rather an ideologue.

    • marcandetty Says:

      The same question could be thrown back at you! Your sources could also be reinterpreted in light of contemporary science, so why don’t you? Probably the same reason the vast majority of the great rabbis have refused to tamper with the meaning of genesis… Because that’s just not what it means and we since we (humans) did not author the Torah it is not ours to mess about with.

  5. Menachem Mevashir Says:

    Marc: Not sure I follow you. What are my sources you refer to? And anyway, you are dodging the question, because since Orthodox Jews do NOT believe that Adam introduced death into the world (ie, they do NOT believe in the Christian notion of “original sin”), then they can very easily explain Genesis to be compatible with evolutionary science.

    I once wrote to Dr. Schroeder to ask if, as an theistic evolutionist, he would agree with Christians who argue that Noah’s Flood was a purely “local event” in order to make it conform to the alleged earth history described by uniformitarianism. He kind of begged off the question and referred me to his books, which I do not own.

    I guess what I am saying is that it seems strange that Jews like Yoram and Gottlieb would embrace a partial literalism with regard to Genesis but not a complete literalism. That is what puzzles me.

    If you believe that Adam DID bring death to the world, why do you reject Original Sin?

  6. Searle Says:

    Dear Menachem,

    Thanks for your response.

    With regards to your comment about finding ways to ‘conform’ Torah to contemporary knowledge, it seems clear to me that you have not yet read Genesis and Genes. Had you done so, it would most likely have obviated this need.

    Can you please explain to me why it is that you don’t consider Rabbi Dr Gottlieb as a scientist? I would have thought that a PhD in Mathematical logic would have been sufficient?

    Anyhow, I think this last issue (re scientists) is besides the point. We are not as interested in what orthodox scientists say, so much as we are interested in what the Torah position is.

    Regards.

  7. Menachem Mevashir Says:

    Did not know R. Gottlieb studied Mathematic logic. Thought it was just ordinary Philosophy, which I don’t consider to be an empirical science. Thanks for information.

    Have not read G&G.

    The Torah position says Adam & Eve brought sin and death into the world. So far as I know, most Orthodox Jews reject that. Only Messianic Jews accept it on face value.

    The Torah also says that if we are righteous and holy, then God will bless our food and water and spare us from disease and illness. Yet most Orthodox Jews quickly run to a doctor for healing while imgesting the unhealthiest array of foods imaginable (at least in Israel, where I lived for many years).

    The Torah describes Levitical cities as having green belts surrounding them. Yet if you walk through Haredi parts of Jerusalem, there is not a single tree or park in sight, but rather a gigantic concrete jungle devoid of green.

    So Jews are quite able and willing to read the Torah selectively. Why in the case of evolution should they be any different?

  8. Searle Says:

    Dear Menachem.

    Thanks for your response.

    My original point remains – read Genesis and Genes. I would be surprised if you still have these questions.

    I don’t have the time to repeat what is written in Genesis and Genes, and would not be able to do half as good a job.

    All the best.

  9. OPERA or Soap Opera? | Torah Explorer Says:

    […] https://torahexplorer.com/2013/04/10/dr-ben-goldacre-and-the-reproducibility-of-research/ […]

  10. Dr. John Ioannidis and the Reality of Research | Torah Explorer Says:

    […] https://torahexplorer.com/2013/04/10/dr-ben-goldacre-and-the-reproducibility-of-research/ […]

  11. Science as a Self-Correcting Mechanism | Torah Explorer Says:

    […] https://torahexplorer.com/2013/04/10/dr-ben-goldacre-and-the-reproducibility-of-research/ […]

  12. movie chattanooga choo choo Says:

    Everything is very open with a precise clarification of the issues.

    It was really informative. Your website is very useful. Many thanks for
    sharing!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: