Science is not a monolith. There are various realms of science, and scientific results slot into a continuum of credibility. Informed consumers of science differentiate between those areas of science in which results are robust and other areas, where results are much more speculative.
In general, science is strongest when it deals with repeatable, observable and limited phenomena. I explained in Genesis and Genes that
The third adjective [i.e. limited] refers to the fact that the phenomenon under study is to some extent isolated; it is more-or-less modular, and so can be studied independently, without having to take account of numerous interactions… studying the climate would not fit into this category because the climate is influenced by a huge number of factors ranging from the local density of trees to the cyclic appearance of sunspots.
In medical research, it is often difficult to study a limited phenomenon. Biological organisms are so complex that there are always myriad factors that need to be taken into consideration. This does not mean that we shouldn’t conduct medical research. It does mean that greater scepticism should be applied by members of the public when assessing scientific claims about medical research than about other, more limited areas of research.
The New York Times recently reported on a 10-year study conducted by a 39-member team that has enormous implications. In this post, I will summarise the article and draw some conclusions about it.
The New York Times begins by stating that “For decades, mice have been the species of choice in the study of human diseases. But now, researchers report evidence that the mouse model has been totally misleading for at least three major killers — sepsis, burns and trauma. As a result, years and billions of dollars have been wasted following false leads.” The study does not claim that mice are useless models for all human diseases. But, its authors said, they do raise troubling questions about diseases like the ones in the study that involve the immune system, including cancer and heart disease.
The paper, published in the Proceedings of the National Academy of Sciences, may explain why every one of nearly 150 drugs tested at huge expense in patients with sepsis has failed. [Sepsis is a frequently-fatal reaction that occurs as the body tries to fight an infection. It afflicts 750 000 patients a year in the United States, kills one-fourth to one-half of them, and costs the nation $17 billion a year. It is the leading cause of death in intensive-care units.] The drug tests were all based on studies in mice. And mice, it turns out, can have something that looks like sepsis in humans, but is very different from the condition in humans.
For more than a year, the research group tried to publish its findings in several journals, without success. [Readers of Genesis and Genes may remember that the crucial paper by Warren and Marshall, detailing the research that would eventually win them a Nobel Prize, was rejected by the Gastroenterological Society of Australia, which deemed it in the bottom 10% of papers submitted.] Crucially, reviewers did not point out scientific errors when rejecting the paper for publication. Instead, the most common response was, ‘It has to be wrong. I don’t know why it is wrong, but it has to be wrong.’ And it’s not as if the results obtained by the researchers were ambiguous. “When I read the paper, I was stunned by just how bad the mouse data are,” Dr. Mitchell Fink, a sepsis expert at the University of California, Los Angeles, said. “It’s really amazing — no correlation at all. These data are so persuasive and so robust that I think funding agencies are going to take note.” Until now, he said, “to get funding, you had to propose experiments using the mouse model.” Furthermore, there were always indications of the shortcomings of the mouse model. One major clue that mice might not really mimic humans in many cases is the well-known fact that it is very hard to kill a mouse with a bacterial infection. Mice need a million times more bacteria in their blood than what would kill a person.
The paper showed that there was no correlation between the genetic responses of mice and those of humans in certain circumstances. One objection of reviewers was that the researchers had not shown the same gene response which had been seen in humans had happened in mice. “They were so used to doing mouse studies that they thought that was how you validate things,” said Ronald W. Davis, a genomics expert at Stanford University and a lead author of the new paper. “They are so ingrained in trying to cure mice that they forget we are trying to cure humans.” But when the group looked for the expected similarities between humans and mice, there were none at all. The drug failures became clear. For example, often in mice, a gene would be used, while in humans, the comparable gene would be suppressed. A drug that worked in mice by disabling that gene could make the response even more deadly in humans.
In Genesis and Genes I quoted the physicist and philosopher Sir John Polkinghorne, who writes that scientists wear theoretical spectacles behind the eyes. Contrary to naive visions of how science is done, scientists do not approach research with a tabula rasa. There is extensive conditioning that impinges on what one sees and, crucially, what one does not see.
I gave several examples of this in Genesis and Genes. One was the discovery of Uranus. Briefly, even though Uranus had been seen by astronomers more than 20 times before it was “discovered” by William Herschel, it was always misidentified as a star. Those who saw Uranus before Herschel “saw” a star because everyone just knew that there are five planets, not six.
This is how two distinguished philosophers of science describe the phenomenon,
Perhaps we may also see here an example of an important human trait, which colors all scientific work and which even the greatest thinkers can never hope to overcome entirely: We all tend to deny the importance of facts or observations not in accord with our convictions and preconceptions, so that sometimes we ignore them altogether, even though, from another point of view, they would stand before our very eyes.
The mice study is a good example of wearing theoretical spectacles behind the eyes. Until now, these spectacles precluded researchers from seeing the limitations of the mouse model of disease, limitations which, from another point of view, now seem to stand before our very eyes.
Medical research is complicated, and involves myriad factors relevant to the pathogen, the human body, and the interaction between the two. One question which is always important to informed consumers of science is, “What assumptions are being made in this research?” In chapter 3 of Genesis and Genes we saw how the assumptions made by Lord Kelvin and colleagues in calculating the age of the world completely misled them. They assumed, for example, that the Earth does not generate any heat. Other than sunshine and the primordial heat left over from the formation of the planet, there is nothing to replenish the Earth’s store of energy. It turned out to be a fatally-flawed assumption once the process of radioactivity was discovered and elucidated, and destroyed Kelvin’s thesis.
In medical research, where mice are the workhorses (work-mice?) of the laboratory, it had been assumed for decades that if mice studies work for some aspects of human physiology and pathology, then they work for all aspects of same. If the new study is correct, this is an assumption that has led to the waste of billions of dollars and has thrown research in these areas off for decades.
In the first chapter of Genesis and Genes, I described the Proof Continuum in some detail. This refers to the spectrum of meanings which the term proof has, depending on the discipline which is discussed. We saw that the term proof has a wide variety of meanings when applied to mathematics, economics, geology, psychology, archaeology, physics and other intellectual disciplines.
One of the truly outrageous claims made by proponents of evolutionary biology is that evolution – not in the limited sense of change over time, but in the broad sense of common descent through natural selection acting on random mutations – is as well established as the fact that the Earth orbits the Sun. It is a statement that ignores basic aspects of the history of science. Even in science’s backyard there are so often paradigm shifts that underline the fallibility of the scientific endeavour. The mice study, if correct, is an example of how, over decades, certain assumptions mislead researchers, even though the subject matter is accessible to direct experimentation. Before this study was published, if anyone had suggested that our model of disease research in mice is possibly seriously flawed, it is very likely that he would have been dismissed with a snort of indignation by medical researchers, snug in their “knowledge” that their models are accurate. We know better now. When claims are made about events that are forever shrouded in the mists of time or space, inaccessible to direct observation and measurement, not corroborated by experiment, never to be repeated and harbouring profound philosophical implications, the correct response is to harbour profound scepticism.
 See http://www.nytimes.com/2013/02/12/science/testing-of-some-deadly-diseases-on-mice-mislead-report-says.html?pagewanted=1&_r=0&pagewanted=all.
Retrieved 28th February 2013.
 Physics, the Human Adventure, Gerald Holton and Stephen G. Brush, Rutgers University Press, 2005, page 8.