Archive for April, 2015

Poor Review

April 17, 2015

The latest issue of Dialogue Magazine contains an essay I submitted, entitled Poor Review. It is reproduced below.

Your life depends on the Krebs cycle.

Even if you haven’t heard of it, your body is powered by the string of chemical reactions named after a German Jewish scientist who was forced to flee his homeland in 1933. In 1953, Hans Krebs was awarded the Nobel Prize in medicine/physiology for his seminal discovery of the eponymous cycle. But there’s a twist to this happy story. In June 1937, when Krebs submitted his crucial paper to Nature, it was returned to him. The paper was eventually published in the obscure journal Enzymologia.

Copernicus’s heliocentric cosmology, Galileo’s mechanics, Newton’s gravity and equations of motion – these ideas never appeared in journal articles. They appeared in books that were reviewed, if at all, by associates of the author. The peer-review process as we know it was instituted after the Second World War, largely due to the huge growth of the scientific enterprise and the enormous pressure on academics to publish ever more papers.

When it comes to peer review of scientific papers, the general public entertains unrealistic, highly-idealized visions of a process by which scientific research is assessed. In theory, peer review is supposed to act as a filter, weeding out the crackpots; in practice, it often turns out to be a way to enforce orthodoxy.

Since the 1950s, peer-review has worked as follows: a scientist wishing to publish a paper in a journal submits a copy of the paper to the editor of a journal. The editor forwards the paper to several academics whom he considers to be experts on the matter, asking whether the paper is worthy of publication. These experts – who usually remain anonymous – submit comments about the paper that constitute the “peer review”. The referees judge the content of the paper on criteria such as the validity of the claims made in the paper, the originality of the work, and whether the work, even if correct and original, is important enough to be worthy of publication. Often, the journal editor will require the author to amend his paper in accordance with the recommendations of the referees.

Prior to the War, university professors were mainly teachers, carrying a teaching load of five or six courses per semester (a typical course load nowadays is one or two courses). Professors with this onerous teaching burden were not expected to write papers. The famous philosopher of science, Sir Karl Popper, wrote in his autobiography that the dean of the New Zealand university where Popper taught during World War II said that he regarded Popper’s production of articles and books a theft of time from the university.

But at some point, universities came to realize that their prestige – and with it the grants they received from governments and corporations – depended more so on the scholarly reputation of their professors than on their teaching skills. And this reputation could only be enhanced through publications. Teaching loads were reduced to allow professors more time for research and the production of papers; salaries began to depend on one’s publication record. Before the War, salaries of professors of the same rank (associate professor, assistant professor, adjunct professor, full professor etc.) were the same (except for an age differential, which reflected experience). Nowadays, salaries of professors in the same department of the same age and rank can differ by more than a factor of two, based on their publication output.

One consequence of all this is that the production of papers has increased by a factor of more than one thousand over the past fifty years. The price that has been paid for this literary fecundity is a precipitous decline in quality. Before the War, when there was no financial incentive to publish papers, scientists wrote them as a labor of love. These days, papers are written mostly to further one’s career. One thus finds that nowadays, most papers are never cited by anyone except their author(s).[1]

Philip Anderson, who won a Nobel Prize for physics, writes that:

“in the early part of the postwar period [a scientist’s] career was science-driven, motivated mostly by absorption with the great enterprise of discovery, and by genuine curiosity as to how nature operates. By the last decade of the century far too many, especially of the young people, were seeing science as a competitive interpersonal game, in which the winner was not the one who was objectively right as [to] the nature of scientific reality, but the one who was successful at getting grants, publishing in Physical Review Letters, and being noticed in the news pages of Nature, Science, or Physics Today… [A] general deterioration in quality, which came primarily from excessive specialization and careerist sociology, meant quite literally that more was worse.”[2]

More is worse. As Nature puts it, “With more than a million papers per year and rising, nobody has time to read every paper in any but the narrowest fields, so some selection is essential. Authors naturally want visibility for their own work, but time spent reading their papers will be time taken away from reading someone else’s.”[3] The number of physicists has increased by a factor of one thousand since the year 1900. Back then, ten percent of all physicists in the world had either won a Nobel Prize or had been nominated for it. Things are much the same in chemistry. The American Chemical Society made a list of the most significant advances in chemistry over the last 100 years. There has been no change in the rate at which breakthroughs in chemistry have been made in spite of the thousand-fold increase in the number of chemists. In the 1960s, United States citizens were awarded about 50 000 patents in chemistry-related areas per year. By the 1980s, the number had dropped to 40 000. But the number of papers has exploded. One result of this publish-or-perish mentality is that groundbreaking papers are often rejected because they are submitted to referees who are incapable or unwilling to recognise novel ideas. Consider these examples:

1. Rosalyn Yalow won the Nobel Prize in Physiology or Medicine in 1977. She describes how her Nobel-winning paper was received: “In 1955 we submitted the paper to Science… the paper was held there for eight months before it was reviewed. It was finally rejected. We submitted it to the Journal of Clinical Investigations, which also rejected it.”[4]

  1. Günter Blobel also won a Nobel Prize in Physiology or Medicine, in 1999. In a news conference given just after he was awarded the prize, Blobel said that the main problem one encounters in one’s research is “when your grants and papers are rejected because some stupid reviewer rejected them for dogmatic adherence to old ideas.” According to the New York Times, these comments “drew thunderous applause from the hundreds of sympathetic colleagues and younger scientists in the auditorium.”[5]
  1. Mitchell J. Feigenbaum thus described the reception that his revolutionary papers on chaos theory received: “Both papers were rejected, the first after a half-year delay. By then, in 1977, over a thousand copies of the first preprint had been shipped. This has been my full experience. Papers on established subjects are immediately accepted. Every novel paper of mine, without exception, has been rejected by the refereeing process. The reader can easily gather that I regard this entire process as a false guardian and wastefully dishonest.”[6]
  1. Theodore Maiman invented the laser, an achievement whose importance is not doubted by anyone. The leading American physics journal, Physical Review Letters, rejected Maiman’s paper on constructing a laser.[7]
  1. John Bardeen, the only person to have ever won two Nobel Prizes in physics, had difficulty publishing a theory in low-temperature solid-state physics that went against the paradigm.[8]
  1. Stephen Hawking needs no introduction. According to his first wife Jane, when Hawking submitted to Nature what is generally regarded as his most important paper on black hole evaporation, the paper was initially rejected.[9] The physicist Frank J. Tipler writes that “I have heard from colleagues who must remain nameless that when Hawking submitted to Physical Review what I personally regard as his most important paper, his paper showing that a most fundamental law of physics called ‘unitarity’ would be violated in black hole evaporation, it, too, was initially rejected.”
  1. Conventional wisdom in contemporary geophysics holds that the Hawaiian Islands were formed sequentially as the Pacific Plate moved over a hot spot deep inside the Earth. This idea was first developed in a paper by the Princeton geophysicist Tuzo Wilson. Wilson writes: “I… sent [my paper] to the Journal of Geophysical Research. They turned it down… They said my paper had no mathematics in it, no new data, and that it didn’t agree with the current views. Therefore, it must be no good. Apparently, whether one gets turned down or not depends largely on the reviewer. The editors, too, if they don’t see it your way, or if they think it’s something unusual, may turn it down. Well, this annoyed me…”[10]

In a paper playfully entitled Not in our Nature but nonetheless published in Nature, Juan Miguel Campanario gives many examples of papers, now considered classics, which were rejected by the world’s most prestigious science journal. Here are two examples:

  1. In 1981, Nature rejected a paper by the British biochemist Robert H. Michell on signalling reaction by hormones. This paper has since been cited more than 1800 times.
  1. A paper by Michael J. Berridge, rejected in 1983 by Nature, ranks at 275 in a list of the most-cited papers of all times, having been cited more than 1900 times.[11]

In 2006, Nature published an essay by Charles G. Jennings, a former editor with the Nature journals and former executive director of the Harvard Stem Cell Institute. As an editor, Jennings was intimately familiar with the peer-review system, and knows full well how badly misunderstood this process is by the public. He writes:

“Whether there is any such thing as a paper so bad that it cannot be published in any peer reviewed journal is debatable. Nevertheless, scientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth.”

Jennings writes that “many papers are never cited (and one suspects seldom read)”. These papers are written, to a large extent, because “To succeed in science, one must climb this pyramid [of journals]: in academia at least, publication in the more prestigious journals is the key to professional advancement.”[12] Advancement, in this context, is measured by career rewards such as recruitment and promotion, grant funding, invitations to speak at conferences, establishment of collaborations and media coverage.

In September 2001, Science reported on a conference on peer review. The article states that traditionally, peer review was considered the “least bad way” to weed out weak manuscripts or research proposals and improve promising ones. But

“that common wisdom was questioned last weekend at a meeting attended by hundreds of editors of medical journals and academics. In a meta-analysis that surprised many – and that some doubt -researchers found little evidence that peer review actually improves the quality of research papers.”

The article continued

“Mention ‘peer review’ and almost every scientist will regale you with stories about referees submitting nasty comments, sitting on a manuscript forever, or rejecting a paper only to repeat the study and steal the glory.”[13]

Sydney Brenner received a Nobel Prize in Physiology or Medicine in 2002, and is revered for his contributions to molecular biology. He was interviewed in February 2014, and the title of the interview is itself illuminating: How Academia and Publishing are Destroying Scientific Innovation.[14] Brenner is fiercely critical of peer review:

“But I don’t believe in peer review because I think it’s very distorted… I think peer review is hindering science. In fact, I think it has become a completely corrupt system. It’s corrupt in many ways, in that scientists and academics have handed over to the editors of these journals the ability to make judgment on science and scientists… Now I mean, people are trying to do something, but I think it’s not publish or perish, it’s publish in the okay places [or perish]. And this has assembled a most ridiculous group of people… I campaigned against this [culture] because I think it is not only bad, it’s corrupt. In other words it puts the judgment in the hands of people who really have no reason to exercise judgment at all. And that’s all been done in the aid of commerce, because they are now giant organisations making money out of it. “

The 2013 Nobel Prize-winning biologist Randy Schekman recently announced that he will no longer submit papers to ‘luxury’ science journals because they corrupt the publication process. Writing in The Guardian, he titled his piece How Journals like Nature, Cell and Science are Damaging Science.[15]

***

The very existence of a blog called Retraction Watch is telling.[16] The site tracks retractions “as a window into the scientific process.” Typical entries, of which there are hundreds, read “Another Nature stem cell paper is retracted”[17] or “University of Texas Southwestern cancer research group notches ninth retraction.”[18] The blog’s editor, medical doctor turned journalist Ivan Oransky, explains that “A retraction means there is something deeply wrong” with a given academic paper. “About two thirds of the time, that’s actually something that’s considered misconduct – the official federal definition of which is falsification, fabrication, or plagiarism.”[19]

But Retraction Watch’s latest scoop made the regular entries pale into insignificance. In July of this year it reported on the busting of a “peer review and citation ring”, causing SAGE Publishing, a major publisher of scientific journals, to retract sixty (!) papers from its Journal of Vibration and Control.[20] Among the scandal’s victims was Taiwan’s education minister, who resigned “to uphold his own reputation and avoid unnecessary disturbance of the work of the education ministry, after the incident ignited a wave of public criticism.”[21] This is what Slate wrote about the debacle:

It may not be entirely fair to liken a “peer review and citation ring” to the academic version of an extortion ring, but there’s certainly fraud involved in both. Retraction Watch, a blog dedicated to chronicling which academic papers have been withdrawn, is reporting that SAGE Publishing, a group that puts out numerous peer-reviewed journals, is retracting 60 papers from its Journal of Vibration and Control after an internal investigation uncovered extensive evidence of severe peer-review fraud.

Apparently researcher Peter Chen, formerly of National Pingtung University of Education in Taiwan, made multiple submission and reviewer accounts – possibly along with other researchers at his institution or elsewhere – so that he could influence the peer review system. When Chen or someone else from the ring submitted a paper, the group could manipulate who reviewed the research, and on at least one occasion Chen served as his own reviewer.[22]

The Washington Post [23] and The Guardian [24] carried similar stories.

***

Why does all of this matter?

First, the narrow point. In Big Science, certain topics – like, say, the weaknesses of evolutionary biology, are well nigh verboten. I hope that by now, the following point is obvious: The claim that scientists critical of evolutionary biology who are not published in Nature, Science et al cannot be accomplished scientists (why, otherwise, haven’t they published there?) is hollow. These journals will not publish material which undermines the cherished paradigm. Contrary to popular misconceptions, Big Science is not open, objective, and tolerant. Even those scientists whose research is done within the paradigm – including virtually all those named above – often find the system to be rigid and even hostile towards those who are truly innovative.

But there’s a broader point to be made as well. I have written and lectured on the interface of Judaism and science for many years. I mostly deal with issues of ultimate origins – of the universe and humanity. These are controversial and fascinating issues, and should be explored dispassionately and in detail. Probably the biggest obstacle I face in trying to convey Judaism’s views on these matters is the public’s distorted view of what scientists are like and what scientific work consists of. Among the general public, there is a pervasive view of scientists as paragons of morality and objectivity. It’s as if the mere fact that one has an aptitude for mathematics or wears a lab coat somehow bestows immunity from human vices.

In Genesis and Genes, I wrote:[25]

The physicist Sir John Polkinghorne is a prominent writer about the intersection of science and theology. In Science and Theology he coins a marvellous phrase:[26]

Scientists do not look at the world with a blank gaze; they view it from a chosen perspective and bring principles of interpretation and prior expectations… to bear upon what they observe. Scientists wear (theoretical) “spectacles behind the eyes”.

Theoretical spectacles behind the eyes. Why does Polkinghorne need to tell his readers that scientists view the world not with a blank gaze, but from a chosen perspective and with prior expectations? Isn’t that only human? The answer:[27]

“Many people have in their minds a picture of how science proceeds which is altogether too simple. This misleading caricature portrays scientific discovery as resulting from the confrontation of clear and inescapable theoretical predictions by the results of unambiguous and decisive experiments… In actual fact… the reality is more complex and more interesting than that.”

The caricature of scientists as supermen is pervasive. The layman is so removed from the experience of actual research that he harbors an image of this profession that is massively distorted. We picture the scientist as popping into existence in the laboratory, like the Greek goddess Athena popping out of Zeus’ head, without any experiences, prejudices, views and background that could impinge on her interpretations and expectations. But scientists are human beings, subject to all the weaknesses, foibles and failings of other human beings.[28]

When I wrote these words in Genesis and Genes, I was focusing on the conditioning to which all scientists are subject and which deeply influences their research. This essay, with its exploration of the mechanics of a crucial component of the scientific edifice – the institution of peer review – is intended as a cautionary note to those who do not understand that scientists are indeed subject to all the weaknesses, foibles and failings of other human beings. They are subject to many forces which influence their research agendas, assumptions and conclusions.

The general public, blissfully shielded from the sludge-like reality of science publishing, takes peer-review to be flawless. Non-experts often mistakenly believe that individual scientists who serve as editors or referees on papers are always open-minded and completely objective in reviewing papers. Peer-review is seen as a gold-standard which guarantees the legitimacy of a paper. Indeed, this mirage is promoted by science propagandists like Jerry Coyne, an evolutionary biologist at the University of Chicago, who wrote in 2005 that “The gold standard for modern scientific achievement is the publication of new results in a peer-reviewed scientific journal.”[29] But the examples in this essay are more than sufficient to demonstrate that the peer-review system is deeply compromised. It is an efficient way of strangling new ideas, rather than a vehicle for promoting truly novel ideas. The peer-review system often stifles true innovation, allowing the reigning paradigm to squash all competition unfairly.

Stephen Jay Gould, one of the most prominent evolutionary biologists of the late 20th Century, sought to dispel the myths promoted by the likes of Coyne when he observed that “[t]he stereotype of a fully rational and objective ‘scientific method,’ with individual scientists as logical (and interchangeable) robots, is self-serving mythology.”[30] The mathematician John Lennox echoed these sentiments, writing that “… the Enlightenment ideal of the coolly rational scientific observer, completely independent, free of all preconceived theories, prior philosophical, ethical and religious commitments, doing investigations and coming to dispassionate, unbiased conclusions that constitute absolute truth, is nowadays regarded by serious philosophers of science (and, indeed, most scientists) as a simplistic myth.”[31] Lawrence K. Altman made a similar point in an article in the New York Times:

“Many nonscientists perceive reviewers to be impartial. But the reviewers, called independent experts, in fact are often competitors of the authors of the papers they scrutinize, raising potential conflicts of interest.”[32]

Professional jealousy is not the only vice involved. Dr. Altman explains that good old-fashioned greed is also a factor. Journals have huge economic interests in preserving the current flawed system, and research scientists play along because peer-reviewed papers are necessary for them to maintain their positions:

The public and many scientists tend to overlook the journals’ economic benefits that stem from linking their embargo policies to peer review. Some journals are owned by private for-profit companies, while others are owned by professional societies that rely on income from the journals. The costs of running journals are low because authors and reviewers are generally not paid.

A few journals that not long ago measured profits in the tens of thousands of dollars a year now make millions, according to at least three editors who agreed to discuss finances only if granted anonymity, because they were not authorized to speak about finances.[33]

***

What’s the solution? Many scientists believe that the system is broken beyond repair. But others, who certainly recognise the ills that plague the peer-review process, believe it’s salvageable. Experiments are being conducted to improve – or sidestep – the current system. For example, some journals no longer grant referees the protection of anonymity. Instead, reviewers are identified and their critiques of papers are made available to the author of the paper being reviewed. The author is then able to defend his paper. This may ameliorate the problem of reviewers who hamper the publication of a paper for less than noble reasons (such as professional jealousy). Another possible solution, already being tried, is the creation of websites that will publish any research paper, whether it is deemed “acceptable” by the establishment or not, as long as it is submitted by credentialed scientists. But this debate, I think, misses the point.

The point is that scientists are human beings. They are fallible; they suffer from professional jealousy, prejudice, greed and every other human failing, to the precise extent that other human beings suffer from these moral maladies. There is nothing in the typical scientist’s training that makes him less vulnerable to subjectivity and prejudice than any other person.

Nothing fundamental is going to change until the science establishment realises that facility with partial differential equations or the ability to fiddle with the knobs and switches of an electron microscope are not guarantors of ratiocination. Character training is just as important. Until the “self-serving mythology” of a scientific method that is impervious to human frailty and is thus rational and objective is recognised as such by scientists and the wider public, and until such time as scientific training incorporates elements of self-analysis and character improvement, one can expect that peer-review, as well as all other elements of modern science, will fall far short of the ideal.

 NOTES:

[1] See the essay by the physicist Frank J. Tipler entitled Refereed Journals: do they insure quality or enforce orthodoxy? The essay appeared in Uncommon Dissent: Intellectuals who find Darwinism Unconvincing, William A. Dembski (editor), ISI Books, 2004. Several of the quotations in this article come from this essay.

[2] Philip Anderson, in Brown, Pais and Pippard, editors, Twentieth Century Physics, American Institute of Physics Press, 1995,page 2029.

[3] http://blogs.nature.com/peer-to-peer/2006/06/quality_and_value_the_true_pur.html

Retrieved 1st June 2014.

[4] Walter Shropshire Jr., editor, The Joys of Research, Smithsonian Institution Press, 1981, page 109.

[5] New York Times, 12th October 1999, page A29.

[6] Mitchell J. Feigenbaum, in Brown, Pais and Pippard, editors, Twentieth Century Physics, American Institute of Physics Press, 1995, page 1850.

[7] Ibid. page 1426.

[8] Lillian Hoddeson, True Genius: The Life and Science of John Bardeen, Joseph Henry Press, 2002, page 300.

[9] Jane Hawking, Music to Move the Stars: A Life with Stephen Hawking, Trans-Atlantic Publications, 1999, page 239.

[10] Walter Shropshire Jr., editor, The Joys of Research, Smithsonian Institution Press, 1981, page 130.

[11] Campanario, J.M. 1993, Not in our Nature, Nature 361:488.

[12] http://blogs.nature.com/peer-to-peer/2006/06/quality_and_value_the_true_pur.html

Retrieved 1st June 2014.

[13] Science 21 September 2001: Vol.293 no. 5538 pp. 2187-2188:

[14] http://kingsreview.co.uk/magazine/blog/2014/02/24/how-academia-and-publishing-are-destroying-scientific-innovation-a-conversation-with-sydney-brenner/

Retrieved 27th May 2014.

[15] http://www.theguardian.com/commentisfree/2013/dec/09/how-journals-nature-science-cell-damage-science.

Retrieved 27th May 2014.

[16] http://retractionwatch.com/

Retrieved 31st July 2014.

[17] http://retractionwatch.com/2014/08/01/another-nature-stem-cell-paper-is-retracted/

Retrieved 4th August 2014.

[18] http://retractionwatch.com/2014/07/31/ut-southwestern-cancer-research-group-notches-ninth-retraction/

Retrieved 4th August 2014.

[19] http://www.forbes.com/sites/billfrezza/2014/07/13/bad-science-muckrakers-question-the-big-science-status-quo/

Retrieved 6th August 2014.

[20] http://retractionwatch.com/2014/07/08/sage-publications-busts-peer-review-and-citation-ring-60-papers-retracted/

Retrieved 4th August 2014.

[21] http://retractionwatch.com/2014/07/14/taiwans-education-minister-resigns-in-wake-of-sage-peer-review-scandal/#more-21533

Retrieved 4th August 2014.

[22]http://www.slate.com/blogs/future_tense/2014/07/08/sage_publishers_journal_of_vibration_and_control_is_retracting_60_papers.html

Retrieved 30 July 2014.

[23] http://www.washingtonpost.com/news/morning-mix/wp/2014/07/10/scholarly-journal-retracts-60-articles-smashes-peer-review-ring/

Retrieved 30 July 2014.

[24] http://www.theguardian.com/media/2014/jul/10/academic-journal-retracts-articles-peer-review-ring

Retrieved 30 July 2014.

 [25] Yoram Bogacz, Genesis and Genes, Feldheim, 2013, pages 10-11.

[26] Science and Theology, John Polkinghorne, Fortress Press, 1998, page 10.

[27] Ibid. page 9.

[28] Dr. Niles Eldredge, a distinguished paleontologist at the American Museum of Natural History in New York, put it this way, “Many scientists really do seem to believe that they have a special access to the truth. They call press conferences to trumpet new discoveries… and they expect to be believed – by their peers, and especially by the public at large. Throwing down scientific thunderbolts from Olympian heights, scientists come across as authoritarian truth givers, whose word must be taken unquestioned.” Speaking as a highly accomplished scientist himself, he unceremoniously shatters this misleading façade: “That all the evidence shows the behavior of scientists to be no different from the ways in which other people behave is somehow overlooked in all this.” See this article http://www.algemeiner.com/2011/06/12/does-dr-niles-eldredge-believe-in-darwinian-evolution/. Retrieved 14th June 2011.

 [29] http://edge.org/conversation/the-case-against-intelligent-design

Retrieved 5th August 2014.

 [30] See Stephen Jay Gould, “In the Mind of the Beholder,” Natural History, Vol. 103 (2):15 (1994). I came across this comment in the following article by Casey Luskin:

 http://www.evolutionnews.org/2012/02/intelligent_des056221.html#fn19

Retrieved 4th August 2014.

 See also

 http://www.evolutionnews.org/2014/07/for_critics_of_087681.html

Retrieved 4th August 2014.

[31] God’s Undertaker: Has Science Buried God? John C. Lennox, Lion, 2009, page 33.

[32] http://www.nytimes.com/2006/05/02/health/02docs.html?pagewanted=2&_r=0

Retrieved 4th August 2014.

[33] Ibid.

Advertisements