In Genesis and Genes, I cited research done by the epidemiologist John Ioannidis:
In a 2005 article in the Journal of the American Medical Association, epidemiologist John Ioannidis showed that among the 45 most highly cited clinical research findings of the past 15 years, 99 percent of molecular research had subsequently been refuted. Epidemiology findings had been contradicted in four-fifths of the cases he looked at, and the usually robust outcomes of clinical trials had a refutation rate of one in four. The revelations struck a chord with the scientific community at large: A recent essay by Ioannidis simply entitled “Why most published research findings are false” has been downloaded more than 100,000 times; the Boston Globe called it “an instant cult classic.”
In 2012 Ioannidis, who is at Stanford, published a paper in Perspectives on Psychological Science. In this post, we will focus on one aspect of that paper viz., the replication of experimental results. Ioannidis points out that contemporary science produces an avalanche of data, with the result that scientists find it difficult to assimilate even a small fraction of the research that is relevant to their area of research. He writes that,
Currently, there are petabytes of scientific information produced on a daily basis and millions of papers are being published annually. [One petabyte is equivalent to one million gigabytes – YB.]
I mentioned this problem in Genesis and Genes:
A paper published in the Proceedings of the National Academy of Scientists in 2006 noted that “More than 5 million biomedical research and review articles have been published in the last 10 years.” That’s an average of 1370 papers per day. And this is just biomedical research.
But the problem is not just volume. Ioannidis points out that raw data is disappearing at gargantuan rates, making it impossible to replicate research, even if one has the inclination and resources to do so:
In most scientific fields, the vast majority of the collected data, protocols, and analyses are not available and/or disappear soon after or even before publication. If one tries to identify the raw data and protocols of papers published only 20 years ago, it is likely that very little is currently available. Even for papers published this week, readily available raw data, protocols, and analysis codes would be the exception rather than the rule. The large majority of currently published papers are mostly synoptic advertisements of the actual research. One cannot even try to reproduce the results based on what is available in the published word.
Ioannidis is not speculating. He is relying on studies that attempt to gauge the repeatability and reproducibility of experiments:
Empirical evidence from diverse fields suggests that when efforts are made to repeat or reproduce published research, the repeatability and reproducibility is dismal (Begley & Ellis, 2012; Donoho, Maleki, Rahman, Shahram, & Stodden, 2009; Hothorn & Leisch, 2011; Ioannidis et al., 2009; Prinz, Schlange, & Asadullah, 2011). Not surprisingly, even hedge funds don’t put much trust on published scientific results (Osherovich, 2011).
This point will be familiar to readers of Genesis and Genes. Recall what I wrote about cancer research:
During a decade as head of global cancer research at Amgen, [Glenn] Begley identified 53 “landmark” publications – papers in top journals, from reputable labs – for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development. Result: 47 of the 53 studies (89%) could not be replicated. He described his findings in a commentary piece published in the journal Nature in March 2012. In a Reuters report, Begley said “It was shocking. These are the studies the pharmaceutical industry relies on to identify new targets for drug development… As we tried to reproduce these papers we became convinced you can’t take anything at face value.” Begley’s experience echoes a report from scientists at Bayer AG. In a 2011 paper titled Believe it or not, they analyzed in-house projects that built on “exciting published data” from basic science studies. “Often, key data could not be reproduced,” wrote Khusru Asadullah, vice president and head of target discovery at Bayer HealthCare in Berlin, and colleagues. Of 47 cancer projects at Bayer during 2011, less than one-quarter could reproduce previously reported findings, despite the efforts of three or four scientists working full time for up to a year. Bayer dropped the projects.
Bayer and Amgen found that the prestige of a journal was no guarantee a paper would be solid. “The scientific community assumes that the claims in a preclinical study can be taken at face value,” Begley and Lee Ellis of MD Anderson Cancer Center wrote in Nature. They and others fear the phenomenon is the product of a skewed system of incentives that has academics cutting corners to further their careers. Part way through his project to reproduce promising studies, Begley met for breakfast at a cancer conference with the lead scientist of one of the problematic studies. “We went through the paper line by line, figure by figure,” said Begley. “I explained that we re-did their experiment 50 times and never got their result. He said they’d done it six times and got this result once, but put it in the paper because it made the best story. It’s very disillusioning.”
A specific example of the above can be found in the context of a subject that I wrote about at length in Genesis and Genes, viz. the Tree of Life, which allegedly demonstrates the relationships between all forms of life on Earth. A new paper in Nature informs us that,
As part of the Open Tree of Life project…, we surveyed publications covering all domains of life and found that most phylogenetic trees and nucleotide alignments from the past two decades have been irrevocably lost.
Bryan T. Drew, working in conjunction with the Open Tree of Life Project, decided to check the data supporting the construction of Darwin’s tree. He writes that,
Of 6,193 papers we surveyed in more than 100 peer-reviewed journals, only 17% present accessible trees and alignments. Contacting lead authors to procure data sets was only 19% successful… We estimate that more than 64% of existing alignments or trees are permanently lost.
The Open Tree of Life website adds more reason for concern. An article entitled “The Glass Is Still Pretty Empty” warns:
Sometimes you wonder whether the glass is half full or half empty. But when it is only filled for four percent – the other 96 percent is just air – there is only one conclusion: it is time for more.
Despite Drew’s estimate in Nature that more than 64% of relationships data are lost, the actual number could be much higher. This article says that only a tiny portion of published trees can be checked against the data:
At least that is what some scientists in the phylogenetic community argue, because only about four percent of all published phylogenies are stored in places such as TreeBASE or Dryad… Several journals in the evolutionary biology field recently adopted policies that encourage or require contributors to make their data publicly available online. Yet, this only leads to the storage of a very small percentage of ten-thousands of phylogenies that have been constructed in the past few decades.
Where replication has actually been tried, it has often failed:
A group of scientists from the Netherlands, United Kingdom, and United States recently published an article about current practices for storing datasets with tree estimates. They concluded that “most phylogenetic knowledge is not easily re-used due to a lack of archiving, lack of awareness of best practices, and lack of community-wide standards for formatting data, naming entities, and annotating data.” As a result, “[m]ost attempts at data re-use seem to end in disappointment.”
As I explained in Genesis and Genes, the public harbours a massively-distorted picture of how contemporary science is done (and in this case, the term public includes science undergraduates). The establishment would have you believe that science is a perfect self-correcting mechanism: experimental results are checked and rechecked; independent teams of researchers verify these results; raw data is available for anyone to examine. Allow me to characterise this vision with a Yiddish term: babkes. Contemporary scientists are swamped by data; they read a minuscule fraction of what is published; virtually nobody has the resources or any incentive to verify others’ research; raw data – the kind that is needed to replicate other laboratories’ results – is lost before anyone else can access it. When replication attempts are made, the rate of success is dismal.
 The original research by Ioannidis can be read here: http://jama.ama-assn.org/content/294/2/218.full.pdf+html. Retrieved 5th June 2011.
 The entire paper can be read here: http://pps.sagepub.com/content/7/6/645.full.pdf.
Retrieved 10th February 2013.
 See http://www.pnas.org/content/103/13/4940.full.pdf+html. Retrieved 23rd July 2011.
 Another study estimated that in 2006, 1.35 million scientific articles were published in 23 750 journals [See http://informationr.net/ir/14-1/paper391.html. Retrieved 15th September 2011.] For an introduction to the appalling consequences of such a flood of results, see the excellent article by the distinguished British pharmacologist David Colquhoun, entitled Publish-or-perish: Peer review and the corruption of science. The article was published in The Guardian on Monday, 5th September 2011 and is available online at http://www.guardian.co.uk/science/2011/sep/05/publish-perish-peer-review-science [Retrieved 15th September 2011]. The author laments that “Pressure on scientists to publish has led to a situation where any paper, however bad, can now be printed in a journal that claims to be peer-reviewed.” Colquhoun explains that
The blame for this sad situation lies with the people who have imposed a publish-or-perish culture, namely research funders and senior people in universities. To have “written” 800 papers is regarded as something to boast about rather than being rather shameful. University PR departments encourage exaggerated claims, and hard-pressed authors go along with them.
The author proceeds to list a few examples of the failure of the peer-review system to ensure robust and accurate journal content. He argues that part of the reason for the lapse in academic publication standards is the pressure on academics to publish many papers. He concludes that frequent publication of results should call into question, rather than enhance, one’s credibility as a diligent and focused researcher.
 See http://www.newsdaily.com/stories/bre82r12p-us-cancer/. Retrieved 31st March 2012.
 See http://www.nature.com/nature/journal/v493/n7432/full/493305f.html.
Retrieved 10th February 2013.
 See http://opentreeoflife.org/2012/11/13/small-portion-of-phylogenetic-data-is-stored-publicly/. Retrieved 10th February 2013.