In the post Dr. Ben Goldacre and the Reproducibility of Scientific Research, I discussed a systemic problem within contemporary science, viz. publication bias. Not all results of scientific research are published; results that stray uncomfortably far from sundry paradigms are sometimes not even submitted by their authors to journals.
A reader objected to this, citing the OPERA experiment as an example of negative results being fearlessly published. Matt wrote:
But I can provide you with countless examples of the researchers deciding to put their results out to the larger community anyway… even at the risk of humiliation if they are found to have messed up. For a recent example, look up the “superluminal neutrino” results from the OPERA experiment.
I didn’t need to look up the OPERA episode, being very familiar with it. But that sorry affair has little in common with the theme of the post Dr. Ben Goldacre and the Reproducibility of Scientific Research, as will be seen in this post.
OPERA stands for Oscillation Project with Emulsion tRacking Apparatus. In September 2011, the experiment electrified the world with the announcement that superluminal neutrinos – subatomic particles that travel faster than light – had been discovered. Physicists usually respond to such grand claims with a laconic “Important, if true.” In this case, had the results been correct, they would not just be important; they would “kill modern physics as we know it”, as Laura Patrizii, leader of OPERA’s Bologna group, put it. The story ended ignominiously, if predictably. The results were found to be incorrect, partly due to some loose cables. But let’s begin at the beginning.
Modern physics is often done by large groups of scientists working together. When you have large collaborations like the OPERA group, it’s prudent to seek consensus before making announcements about the research. Dmitri Denisov, a physicist at Fermilab in Batavia, Illinois, says it is standard procedure to wait to publish a paper until everyone in the collaboration has signed on. “We really strive to have full agreement,” he says. “In some cases it takes months, sometimes up to a year, to verify that everyone in the collaboration is happy.” In the case of OPERA, 15 of the 160 members refused to add their names to the original paper because they felt the announcement and submission of the results for publication were premature. “I didn’t sign because I thought the estimated error was not correct,” said team member Luca Stanco of the National Institute of Nuclear Physics in Italy. In a New Scientist article, Stanco was quoted as saying that “We should have been more cautious, more careful, presented the result in not such a strong way, more preliminarily. Experimentalists in physics can make mistakes. But the way in which we handle them, the way we present them – we have some responsibility for that.” Physics World mentioned Caren Hagner, leader of the OPERA group at Hamburg University and one of the people whose name did not appear on the pre-print. She too argued that the collaboration should have carried out the extra checks before submitting the paper for peer review.
The OPERA operatives were in such a rush to announce their scoop to the world that they failed to apply basic prudence. Janet Conrad, a particle physicist at MIT, said that much of the negative reaction from the physics establishment to the announcement stemmed from the fact that there were insufficient experimental checks carried out prior to the announcement. “A [paper in] Physical Review Letters is four pages long. An experiment is vastly more complicated than that,” she says. “So we have to rely on our colleagues having done all of their cross checks. We don’t expect to make a retraction within a year.” Fermilab’s Joseph Lykken concurred. “Precisely because these are big, complicated experiments, the collaborations have a responsibility to both the scientific community and to the taxpayers to perform due diligence checks of the validity of their results,” he said. “The more surprising the result, the more time one must spend on validation. Anyone can make a mistake, but scientific collaborations are supposed to catch the vast majority of mistakes through internal vetting long before a new result sees the light of day.” CERN physicist Alvaro De Rujula also had strong words in this regard. “The theory of relativity is exquisitely well-tested and consistent. Superluminal neutrinos were far, far too much in violation of the rules to be believed, even for a nanosecond. That ought to have made the OPERA management have everything checked even more carefully. Alas, it turned out not to be a subtle error, but mainly a bad connection, the very first thing one checks when anything misbehaves.”
Then, again in violation of good practice, the OPERA results were announced to the press rather than first presented to peers through the usual science channels. The physicist Lawrence M. Krauss, director of the Origins Project at Arizona State University (and an ardent atheist) authored an op-ed in the Los Angeles Times entitled Colliding Theories, with the subtitle Findings that showed faster-than-light travel were released too soon. He wrote,
What is inappropriate, however, is the publicity fanfare coming before the paper has even been examined by referees. Too often today, science is done by news release rather than waiting for refereed publication.
What makes all of this even more surprising is that the OPERA collaboration did not have a direct competitor from which a scoop had to be snatched. The physicists were in a position to carefully check and re-check their results before rushing off to make their announcement.
As a result of the fiasco, OPERA spokesman Antonio Ereditato of the University of Bern in Switzerland and experimental coordinator Dario Autiero of the Institute of Nuclear Physics in Lyon, France, resigned following a 16-13 no-confidence vote from the collaboration’s other leaders. An indication of just how embarrassing this episode was for physics is that CERN (European Centre for Nuclear Research), the European collaboration that supplied the neutrinos to the OPERA experiment, had no official comment on the resignations, distancing itself from the OPERA experiment despite its central role in publicizing the original results. Physics World reported that a press officer for CERN refused to be identified and emphasised that OPERA was “not a CERN collaboration” since it “only sends [OPERA] a beam of neutrinos.”
To some extent, the OPERA debacle was about grabbing headlines. As one report put it,
If faster than light neutrinos do exist, there need to be many rounds of testing, independent analyses and rigorous peer review before we can start announcing dents in Einstein’s bedrock theories. But, as is abundantly clear in this world of fierce media competition, social media and science transparency, any theory is a good theory so long as it makes a good story — as long as the scientific method has been followed and the science is correctly represented by the writer, that is. [Italics in the original.]
Let us digress for a moment to discuss a few points that are relevant to material discussed in Genesis and Genes, before returning to the topic of publication bias.
I explained in Genesis and Genes that the public almost always misunderstands what is meant by “measurement” in the context of contemporary science. Measurements in cosmology and physics do not mean that someone is doing something as prosaic and straightforward as reading a temperature off a thermometer. The procedure is far more complicated, and introduces enormous amounts of complexity into the endeavour. This is something that Professor Krauss stressed in the article he penned for the LA Times:
The claim that neutrinos arrived at the Gran Sasso National Laboratory in Italy from CERN’s Large Hadron Collider in Switzerland on average 60 billionths of a second before they would have if they were traveling at light speed relies on complicated statistical analysis. It must take into account the modeling of the detectors and how long their response time is, careful synchronization of clocks and a determination of the distance between the CERN accelerator and the Gran Sasso detector accurate to a distance of a few meters. Each of these factors has intrinsic uncertainties that, if misestimated, could lead to an erroneous conclusion.
Informed consumers of science realise that words like measure – which convey a high degree of certainty to the public – in reality reflect something far murkier. This is why, in the post Missing Mass, I pointed out that cosmology is much more theory than observation. The public, inasmuch as it knows anything about the expansion of the universe, for example, entertains fantasies about astronomers watching galaxies flying off into the cosmic sunset, like an airplane slowly moving across the distant horizon. That’s nonsense. To “measure” the expansion of the universe, inferences are made on the basis of complex statistical analyses which depend on layer upon layer of assumption and analysis. In Genesis and Genes, I discussed the work of the brilliant mathematician and member of the National Academy of Sciences Irving Segal. I wrote that,
The most recent study by Segal and his colleagues contained a detailed analysis of Hubble’s law based on data from the de Vaucoleurs survey of bright cluster galaxies, which includes more than 10 000 galaxies. (It is worthwhile noting that Edwin Hubble’s own analysis was based on a puny sample of twenty galaxies.) The results are astounding. The linear relationship that Hubble saw between redshift and apparent brightness could not be seen by Segal and his collaborators. “By normal standards of scientific due process,” Segal wrote, “the results of [Big Bang] cosmology are illusory.”
The debate between Segal and his detractors was not about who had more acute eyesight; it was about ultra-complex models and statistical analysis. This should give informed consumers of science pause when they encounter reports of “measurements” in cutting-edge science.
I pointed out in Genesis and Genes that there exists a misconception of science as the ultimate cosmopolitan pursuit, devoid of any nationalistic flavour which might influence research. The truth is that, being a human endeavour, such factors do influence scientific research. Remember the part about acupuncture? I wrote,
Between 1966 and 1995, there were forty-seven studies of acupuncture in China, Taiwan, and Japan, and every trial concluded that acupuncture was an effective medical treatment for certain conditions. During the same period, there were ninety-four clinical trials of acupuncture in the United States, Sweden, and the United Kingdom, and only fifty-six per cent of these studies found any therapeutic benefits.
Controlled, double-blind clinical trials are not magic bullets. One’s cultural background influences research, and this was a factor in OPERA also. One news item states that “The large international collaboration has had to contend not just with the usual personality conflicts, but also with cultural differences between Italian, Northern European, and Japanese scientists. The added scrutiny from the controversial result exacerbated those tensions.”
At any rate, the OPERA experience has little to do with ordinary, day-to-day publication bias. Once OPERA produced a tsunami of publicity with its premature announcement of superluminal neutrinos, its leaders had no choice but to come clean about the various failures that plagued their experiment. Back on the ranch, far from the limelight, the fact is that uncomfortable results are often just ignored, exiled to distant directories in one’s hard-drive. They don’t make headlines; they don’t provoke resignations; they just don’t get reported and published. And that produces a distorted picture in the minds of scientists and the public with regards to important issues. In the article by Professor Krauss in the LA Times, Kraus – who is an enthusiastic adherent of scientism – writes that
What is inappropriate, however, is the publicity fanfare coming before the paper has even been examined by referees. Too often today, science is done by news release rather than waiting for refereed publication. Because a significant fraction of experimental results ultimately never get published or are not later confirmed, providing unfiltered results to a largely untutored public is irresponsible. [Emphasis added.]
One can quibble with Krauss regarding how much filtering – this is a synonym for prejudice – must be done to protect the public from unorthodox research findings. But the fact is that a significant portion of research is never published. One reason for this is that researchers are trapped in paradigms that stain certain research results as wrong. As Nottingham University astronomer Michael Merrifield explains,
And, more worrying, is something that scientists like to push under the carpet… there’s psychology in this as well. If, in 1985, I made a measurement of the distance [from the Sun] to the centre of the galaxy when everyone said it was ten kilo-parsecs, and I got an answer that said it was seven kilo-parsecs, I would have thought, “Well, I must have done something wrong” and I would have stuck it in some filing cabinet and forgot about it; whereas if I had got an answer that agreed with the consensus, I’d probably have published it… In this error process, there’s also psychology. As I say, scientists are very uncomfortable about this, because we have this idea that what we are doing is objective and above such things. But actually, there is a lot of human interaction and psychology in the way we do science.
Some in the science establishment try to avoid confronting this reality by invoking ideal worlds, in which various safeguards eliminate any residual doubt from experiments. But scientific research – like virtually all human activity – is more ambiguous than these scientists would have you believe. In Genesis and Genes I quoted the physicist and philosopher Sir John Polkinghorne:
Many people have in their minds a picture of how science proceeds which is altogether too simple. This misleading caricature portrays scientific discovery as resulting from the confrontation of clear and inescapable theoretical predictions by the results of unambiguous and decisive experiments… In actual fact… the reality is more complex and more interesting than that.
Nobody doubts that there are many sincere politicians out there. And nobody denies that there is a gaping gulf between election-season promises and post-election reality. After the debris of elections is cleared and the votes tallied, the real, gritty, grey world of horse-trading, budgetary constraints, political alliances and a host of other factors intervene to make politicians, well, politicians.
Science – including the realm of the hard sciences – is a human endeavour. Scientific research is subject to a galaxy of factors beyond the nuts and bolts of the laboratory. It is affected by every condition related to human nature. OPERA is a good example of an experiment going awry because of mundane weaknesses such as impulsivity, the pursuit of glory and bad judgment. But the fact that scientific research happens in the real world and not in some idealized version thereof is just as true in the day-to-day research that never makes headlines.
Informed consumers of science recognise this, and recognise the limitations that these weaknesses impose upon the credibility of scientific research. Science is strong – though never infallible – when it explores phenomena that are repeatable, observable and limited. Its credibility diminishes rapidly as it meanders from these parameters. And when science makes absolute statements about the history of the universe or life, you should take them with a sack of salt.
The post Dr. Ben Goldacre and the Reproducibility of Research:
The post Missing Mass:
The quotations about OPERA in this post come from the following sources:
Retrieved 21st April 2013.
Professor Merrifield can be watched here: