Replication of Experimental Data

In Genesis and Genes, I cited research done by the epidemiologist John Ioannidis:

In a 2005 article in the Journal of the American Medical Association, epidemiologist John Ioannidis showed that among the 45 most highly cited clinical research findings of the past 15 years, 99 percent of molecular research had subsequently been refuted. Epidemiology findings had been contradicted in four-fifths of the cases he looked at, and the usually robust outcomes of clinical trials had a refutation rate of one in four. The revelations struck a chord with the scientific community at large: A recent essay by Ioannidis simply entitled “Why most published research findings are false” has been downloaded more than 100,000 times; the Boston Globe called it “an instant cult classic.”[1]

In 2012 Ioannidis, who is at Stanford, published a paper in Perspectives on Psychological Science.[2] In this post, we will focus on one aspect of that paper viz., the replication of experimental results. Ioannidis points out that contemporary science produces an avalanche of data, with the result that scientists find it difficult to assimilate even a small fraction of the research that is relevant to their area of research. He writes that,

Currently, there are petabytes of scientific information produced on a daily basis and millions of papers are being published annually. [One petabyte is equivalent to one million gigabytes – YB.]

I mentioned this problem in Genesis and Genes:

A paper published in the Proceedings of the National Academy of Scientists in 2006 noted that “More than 5 million biomedical research and review articles have been published in the last 10 years.” That’s an average of 1370 papers per day.[3] And this is just biomedical research.[4]

But the problem is not just volume. Ioannidis points out that raw data is disappearing at gargantuan rates, making it impossible to replicate research, even if one has the inclination and resources to do so:

In most scientific fields, the vast majority of the collected data, protocols, and analyses are not available and/or disappear soon after or even before publication. If one tries to identify the raw data and protocols of papers published only 20 years ago, it is likely that very little is currently available. Even for papers published this week, readily available raw data, protocols, and analysis codes would be the exception rather than the rule. The large majority of currently published papers are mostly synoptic advertisements of the actual research. One cannot even try to reproduce the results based on what is available in the published word.

Ioannidis is not speculating. He is relying on studies that attempt to gauge the repeatability and reproducibility of experiments:

Empirical evidence from diverse fields suggests that when efforts are made to repeat or reproduce published research, the repeatability and reproducibility is dismal (Begley & Ellis, 2012; Donoho, Maleki, Rahman, Shahram, & Stodden, 2009; Hothorn & Leisch, 2011; Ioannidis et al., 2009; Prinz, Schlange, & Asadullah, 2011). Not surprisingly, even hedge funds don’t put much trust on published scientific results (Osherovich, 2011).

This point will be familiar to readers of Genesis and Genes. Recall what I wrote about cancer research:

During a decade as head of global cancer research at Amgen, [Glenn] Begley identified 53 “landmark” publications – papers in top journals, from reputable labs – for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development. Result: 47 of the 53 studies (89%) could not be replicated. He described his findings in a commentary piece published in the journal Nature in March 2012. In a Reuters report, Begley said “It was shocking. These are the studies the pharmaceutical industry relies on to identify new targets for drug development… As we tried to reproduce these papers we became convinced you can’t take anything at face value.”[5] Begley’s experience echoes a report from scientists at Bayer AG. In a 2011 paper titled Believe it or not, they analyzed in-house projects that built on “exciting published data” from basic science studies. “Often, key data could not be reproduced,” wrote Khusru Asadullah, vice president and head of target discovery at Bayer HealthCare in Berlin, and colleagues. Of 47 cancer projects at Bayer during 2011, less than one-quarter could reproduce previously reported findings, despite the efforts of three or four scientists working full time for up to a year. Bayer dropped the projects.
Bayer and Amgen found that the prestige of a journal was no guarantee a paper would be solid. “The scientific community assumes that the claims in a preclinical study can be taken at face value,” Begley and Lee Ellis of MD Anderson Cancer Center wrote in Nature. They and others fear the phenomenon is the product of a skewed system of incentives that has academics cutting corners to further their careers. Part way through his project to reproduce promising studies, Begley met for breakfast at a cancer conference with the lead scientist of one of the problematic studies. “We went through the paper line by line, figure by figure,” said Begley. “I explained that we re-did their experiment 50 times and never got their result. He said they’d done it six times and got this result once, but put it in the paper because it made the best story. It’s very disillusioning.”

A specific example of the above can be found in the context of a subject that I wrote about at length in Genesis and Genes, viz. the Tree of Life, which allegedly demonstrates the relationships between all forms of life on Earth. A new paper in Nature informs us that,

As part of the Open Tree of Life project…, we surveyed publications covering all domains of life and found that most phylogenetic trees and nucleotide alignments from the past two decades have been irrevocably lost.

Bryan T. Drew, working in conjunction with the Open Tree of Life Project, decided to check the data supporting the construction of Darwin’s tree. He writes that,

Of 6,193 papers we surveyed in more than 100 peer-reviewed journals, only 17% present accessible trees and alignments. Contacting lead authors to procure data sets was only 19% successful… We estimate that more than 64% of existing alignments or trees are permanently lost.[6]

The Open Tree of Life website adds more reason for concern.[7] An article entitled “The Glass Is Still Pretty Empty” warns:

Sometimes you wonder whether the glass is half full or half empty. But when it is only filled for four percent – the other 96 percent is just air – there is only one conclusion: it is time for more.

Despite Drew’s estimate in Nature that more than 64% of relationships data are lost, the actual number could be much higher. This article says that only a tiny portion of published trees can be checked against the data:

At least that is what some scientists in the phylogenetic community argue, because only about four percent of all published phylogenies are stored in places such as TreeBASE or Dryad… Several journals in the evolutionary biology field recently adopted policies that encourage or require contributors to make their data publicly available online. Yet, this only leads to the storage of a very small percentage of ten-thousands of phylogenies that have been constructed in the past few decades.

Where replication has actually been tried, it has often failed:

A group of scientists from the Netherlands, United Kingdom, and United States recently published an article about current practices for storing datasets with tree estimates. They concluded that “most phylogenetic knowledge is not easily re-used due to a lack of archiving, lack of awareness of best practices, and lack of community-wide standards for formatting data, naming entities, and annotating data.” As a result, “[m]ost attempts at data re-use seem to end in disappointment.”

As I explained in Genesis and Genes, the public harbours a massively-distorted picture of how contemporary science is done (and in this case, the term public includes science undergraduates). The establishment would have you believe that science is a perfect self-correcting mechanism: experimental results are checked and rechecked; independent teams of researchers verify these results; raw data is available for anyone to examine. Allow me to characterise this vision with a Yiddish term: babkes. Contemporary scientists are swamped by data; they read a minuscule fraction of what is published; virtually nobody has the resources or any incentive to verify others’ research; raw data – the kind that is needed to replicate other laboratories’ results – is lost before anyone else can access it. When replication attempts are made, the rate of success is dismal.

References:

[1] The original research by Ioannidis can be read here: http://jama.ama-assn.org/content/294/2/218.full.pdf+html. Retrieved 5th June 2011.

[2] The entire paper can be read here: http://pps.sagepub.com/content/7/6/645.full.pdf.
Retrieved 10th February 2013.

[3] See http://www.pnas.org/content/103/13/4940.full.pdf+html. Retrieved 23rd July 2011.

[4] Another study estimated that in 2006, 1.35 million scientific articles were published in 23 750 journals [See http://informationr.net/ir/14-1/paper391.html. Retrieved 15th September 2011.] For an introduction to the appalling consequences of such a flood of results, see the excellent article by the distinguished British pharmacologist David Colquhoun, entitled Publish-or-perish: Peer review and the corruption of science. The article was published in The Guardian on Monday, 5th September 2011 and is available online at http://www.guardian.co.uk/science/2011/sep/05/publish-perish-peer-review-science [Retrieved 15th September 2011]. The author laments that “Pressure on scientists to publish has led to a situation where any paper, however bad, can now be printed in a journal that claims to be peer-reviewed.” Colquhoun explains that

The blame for this sad situation lies with the people who have imposed a publish-or-perish culture, namely research funders and senior people in universities. To have “written” 800 papers is regarded as something to boast about rather than being rather shameful. University PR departments encourage exaggerated claims, and hard-pressed authors go along with them.

The author proceeds to list a few examples of the failure of the peer-review system to ensure robust and accurate journal content. He argues that part of the reason for the lapse in academic publication standards is the pressure on academics to publish many papers. He concludes that frequent publication of results should call into question, rather than enhance, one’s credibility as a diligent and focused researcher.

[5] See http://www.newsdaily.com/stories/bre82r12p-us-cancer/. Retrieved 31st March 2012.

[6] See http://www.nature.com/nature/journal/v493/n7432/full/493305f.html.
Retrieved 10th February 2013.

[7] See http://opentreeoflife.org/2012/11/13/small-portion-of-phylogenetic-data-is-stored-publicly/. Retrieved 10th February 2013.

Advertisements

16 Responses to “Replication of Experimental Data”

  1. Matt Says:

    It seems to me that you spend a lot of time arguing that science is fallible. Of course it is. So is every human endeavor. You also spend a lot of time appealing to metascientific arguments based on science history, sociology, and philosophy of science. True as these sorts of arguments may be (to some extent):

    If the truth is on your side, shouldn’t you be able to argue your position purely on the basis of the science, without needing to resort to innuendo and metascientific arguments?

    It seems to me that those who successfully challenge the “scientific orthodoxy” ultimately win because the evidence is on their side, not because they accuse their colleagues of bias, or quote Thomas Kuhn, or making exhaustive lists of their rivals scientific blunders…

    If you are challenging a specific claims in science, I personally would prefer that you address reproducibility of *those claims*, rather than speak in vague generalities about replication of research in science as a whole. Such generalities are indeed important to those of us who care about the science l’shma. But, they strike me as a distraction, with regard to the points you are trying to make.

    As I like to say to non-scientists: It is very easy to cast doubt on an idea. True scientific skepticism is much harder.

  2. DD Says:

    Matt,

    Allow me to step in for Yoram for a moment (uh, permission pending, of course…). First of all, it is actually quite necessary to emphasize and re-emphasize that we are not dealing with just a fuzzy “oh, science is sometimes fallible” type of feeling which can then be easily dismissed in the face of an oversaturated media hype about how certain, in fact, these studies are. We see that the certainy is almost entirely illusory, that the results are not reproducible, that papers are syntheses of previous papers and not independent confirmations, etc. This weakens the claim that those who wish to be consistent *must* reconcile the Torah account with the popular scientific account. It’s not quite a must. It’s more like a may, or a could.

    Secondly, this is not innuendo. It is merely the basis for framing the debate properly. When he actually wants to discuss specific points, Yoram certainly does so. Consider his discussions of homology, evolutionary embryology, the lack of speciation or even significant mutation or formative reduplication in E. Coli and d. melanogaster, contradictory results of genetic phylogeny in different proteins, etc.

    All this is not to say that he is definitely correct about everything (beacuse somehow I see you saying “Yeah, but I disagree with all that! Phooey!”), merely that he puts forward the kind of specific information you describe. It is not a vague “casting doubt.” When homology is touted as a great proof for the evolutionary process and homologous structures are traced to unrelated genes, this is a potential refutation of the proof, not an unwarranted “doubt.”

    Now even so, I actually would also have preferred a bit more on actual inconsistencies in phylogeny, and I would agree with Matt that there is, perhaps, a bit of overemphasis on general flaws (my first paragraph notwithstanding). But much of this depends on the kind of audience Yoram is expecting, whether he is writing with first-time readers in mind, etc. So I can’t judge, I guess.

    By the way, I, too, get quite annoyed when all there is to go on is Thomas Kuhn (as great as his work may be). His book should be a cornerstone of science education, but shouldn’t be invoked when discussing, say, a T-test or electron normalization.

    • Matt Says:

      Hi DD,

      Thanks for the well-written response.

      For the record, I’m a big fan of Kuhn. But, his arguments are often stretched pretty far by some critics of science. Moreover, I know many people with a very superior grasp of history and philosophy of science who would fundamentally disagree with Yoram’s claims and approach on this point. Thanks for seconding that sentiment.

      I disagree with the general argument presented in this article and particularly with your claim that “the certainty [in science] is almost entirely illusory”. I do not think that one can make general claims that apply equally to all fields of science. For example, the drug research community faces vastly different problems than, say, particle physics (my field). Even where these replication problems (described in this post) do apply, it is not necessarily the case that they outright discredit the major truth-claims of the field. A lot of the research that goes without replication is often uninteresting and tangential to the big questions. Much as I would like to see *all* research duplicated, I find it more important that the *key findings* are reproduced. Given limited resources, what matters as that he community demonstrates an ability to identify the important research and reproduce *those findings*. I feel that this generally *is* the case, with perhaps some exceptions.

      In short, there is a tremendous amount of variation from field-to-field in science. There is also a tremendous amount of nuance in the question of how these problems impact their respective fields. To allow vague generalization of these points as a refutation of specific research is indeed to rely on innuendo.

      Some of the problems Yoram addresses in this article are very new and thoughtful people are working hard to address them. Yoram quotes the “Tree of Life” project as an example of weakness, but I see strength in it. A problem was identified and people are working to correct it. I think they will succeed in their effort, and sooner than Yoram would like them to. I have seen other fields of science make similar transformations from being a mess of unavailable raw data to reaching the point where anyone can access raw data, code, and the documentation used in every major publication.

      In short, if Yoram wants to A that Finding A has not been reproduced then I think he has a serious claim to make. But, absolutely *is* innuendo for him to cast doubt upon Finding A, purely on the basis of his general arguments above.

      >>
      “This weakens the claim that those who wish to be consistent *must* reconcile the Torah account with the popular scientific account. It’s not quite a must. It’s more like a may, or a could.”
      >>

      Herein I think lies the problem. And, on this point you and I disagree less than you think. You take issue with scientific arguments made “by authority” and I have no problem with that. I don’t think Torah must reconcile science simply because it is claimed with scientific authority.

      However, I believe strongly that Torah should engage scientific research objectively and honestly. In my own experience, I have been disappointed to discover a sub-set of the Jewish world which questions the motives and intelligence of those of us who, having made educated evaluations of the scientific evidence, feel that it is indeed compelling. I have a problem with rhetoric that accuses the scientific community of nefarious atheistic motivations, and that casts doubt on the science based on misrepresentations of the research. I am shomer mitvos myself, and some of the things I found in the last few years have really bothered me and challenged my faith. I do not purport to be an expert on the subject of evolution (to my knowledge neither is Yoram), but I have enough exposure with the science and with the scientists to feel that the picture painted on this blog is unfair and misrepresentative.

      I find articles like this one to be a distraction. I personally feel that the scientific evidence for evolution is compelling, and that the last ten years have brought some of the most compelling evidence yet. That the Universe is older than 6,000 years is -in my own view- impossible to argue against, without at least claiming the Hashem intentionally planted evidence to make it look so. I would love to discuss the science behind these points. And, again, if the science is on Yoram’s side then we can focus on the veracity of *those specific* findings, and not concern ourselves with tangential issues of whether or not drug research is reproducible.

  3. Matt Says:

    I should just add (with much appreciation): The discussions forum on this blog is very civil. Yoram always posts my comments and often responds. We can disagree, but I appreciate the civility and openness.

  4. DD Says:

    Matt,

    I’m not going to use the Reply option because the narrower columns are not suited for longer replies.

    Your reply is divided into two separate parts. One of them deals with what I feel is the main content of this post. The second one deals with an important meta-issue surrounding the approach to scientific findings in the religious Jewish community. I’m dividing my reply similarly.

    Let us take my statement that certainty is illusory. It is philosophically incorrect, as it has been from the days of Aristotle, to imagine that the scientific method is sound as a syllogism. This is not true and never has been, for the simple reason that we cannot effectively separate cause from correlation in unobserved reality (and unobserved reality is what science is about–when it becomes observed, it’s just observation). What we do is continuously refine our ideas to bring them in closer accord with the world around us, while recognizing that no amount of fine-tuning will ever guarantee us anything.

    As such, we are all pragmatists. We take Aspirin for our headaches and have surgery for our hearts because the risk is statistically low enough and the benefit, conversely, high enough. Thirty years from now, when Aspirin is found to cause arterial sclerosis, I will stop taking it. I’m being facetious, obviously, but this can be applied directly into all scientific progress.

    Any pronouncement, therefore, that we have certainty that extends beyond the realm of the practical, is bogus. No reason to accept it whatsoever. Sure, it might be true, and the way it looks now it fits a lot of information, but an empirical proof is not a mathematical one, and as such is subject to (literally) an infinite amount of weaknesses. (I don’t mean to invoke New Age literature here, but I’ll just note that this is one of the things that supposedly drove Robert Pirsig crazy.) The opinion on the age of the earth circa 1900 is an outstanding example of this problem.

    Thus, when people try to reach back into history and pronounce that they know, with a certainty which can only be described as bordering on epistemological, that constants a, b, and c were the same as they are now, based on solutions to equations which all suffer from the same assumptions, and that animals x, y, and z had this precise arrangement of organs, we must not imagine that this comes with the same certainty as a divine pronouncement. It’s gone beyond the pragmatic truth which is the realm of science. This is one of the things I intended by my phrase. (The question of whether or not we have divine pronouncements on this matter is something else altogether.)

    The second thing I intended is that even on the pragmatic level, certainty is sometimes wanting. In this, I was referring not specifically to this post, but to all of Yoram’s “general” writings which indicate just how “certain” we should assume an idea is when it is touted as such by some spokesman or publication. Of course it’s possible that all scientists with a brain agree to NDE, but it’s also quite possible, as we now know, that this is a fib. I think you are mistaken in assuming that Yoram is always dealing directly with refutation (although he does do that). More often than not I think he is sounding a word of caution, as in this post.

    Concerning the second half of your reply, I actually have nothing to write. What can I say? I am also very troubled by the reactionary approach. I live in the epicenter of that attitude, and it’s not easy. As a practical behavior, I remind myself of the many learned people I know within even this world that are not like that, and that are willing to consider more than they might have been taught as children. I personally don’t count Yoram among those who reject all scientific efforts, as I think he makes clear throughout his book.

    I myself am in the process of sifting through the data in many areas. I usually do not even open popular books, but Yoram’s seemed to me better written and more thoroughly researched than the average one, so I got it. I would also be very interested in discussing all these issues in person with someone who is also in the process of going through them. I don’t know if it’s practical, as I live in New Jersey and you might be just about anywhere.

    • Sam Says:

      First of all I would like to thank DD and Matt for this very informative, clear and civil dialogue. It is of great pleasure to read.

      DD, I’m afraid that Matt did not receive notice of your response since you did not “Reply” on his post…

      Thanks again, and I’m looking forward to the continuation of this conversation.

    • Matt Says:

      Hey DD,

      First and foremost, I agree with you that science does not possess certitude.

      But, it is a straw man to say that the scientific community claims certitude and infallibility. Scientists are very honest and open with what we don’t know, which is why Yoram has a very easy time fishing for quotes to support his narrative. So, if you are skeptical of “those scientists” claiming certitude where it does not exist, then you are not listening to what the scientific community is saying as a whole. Reading this blog, it is easy to get the sense that there is a “Church of Darwin”. But, it is wrong and unfair to suggest this.

      And, just because a subject lacks certitude does not mean that it lacks overwhelming evidence. No human endeavor in this world is certain. We have to work with the best evidence we are given. We cannot use a lack of certitude to digress into nihilism. When scientific theories making strongly testable predictions and those predictions come true, I have to place some weight on that. In my own field, we can derive certain fundamental constants to the trillionths place and validate them through measurements! If you want to convince me of a Torah proposition, you need to make logical and evidence based arguments. And the same is true for scientific arguments. And, all I can do is weigh the arguments and the evidence. I can never be certain. I have to go where the evidence takes me.

      Your defense of Yoram’s “general argument”s comes from the sense that he is presenting a skeptical view. I feel that this argumentative style goes beyond skepticism into the realm of “doubt”. The former is an intellectual proposition, while the latter is irrational. Stringing together lot’s of quotes from technical papers to make a rhetorical argument is not how science works. Scientific subjects are complex. It is very easy to present laundry lists of uncertainties out-of-context to make a subject matter look weak. Putting those uncertainties *in context* is very difficult unless you have a strong grasp of the subject matter and unless you do the hard work. It is much easier to make researchers look like fools than it is to honestly assess their work. Be careful of that.

  5. DD Says:

    Matt,

    It appears you’ve done what I’d mentioned people do in these situations: “Yes, science is not certain. But it’s so certain…” This is a fundamental misunderstanding.

    What I mean is that I set up the following train of thought: Since we do not possess the necessary information, we cannot render what we know to be accurate judgments about the mechanics of the world. So everything is “correct till proven incorrect.” When we are given statements from a source that does possess the necessary information, and in fact is that information (I’m using Maimonides’ language in the second chapter of Yesodei HaTorah), we are immediately dealing with a different order of certainty altogether, and so there is simply no contest whatsoever. Your statement, “If you want to convince me of a Torah proposition, you need to make logical and evidence based arguments,” is false, because if we’re using the same terminology, the Torah does not make propositions, it reveals facts. There’s no need for me to convince an inherently imprecise observer of the correctness of accurate observations for them to be accurate.

    The only room for questioning is “Do we have statements from the Torah regarding the world,” and “Is the Torah a document reflecting information from the kind of source that we claim for it.” Here, I think, is the main discussion, and there is plenty of room for it. If these two propositions have been established, however, the discussion is over.

    Regarding the Church of Darwin, we’d have to do a serious sociological study to establish whether or not “the scientific community,” which is an undefined element at the moment as far as I’m concerned, subscribes to such ideas. You claim they do not. Okay. My father’s experience as a professor of mathematics, and that of his colleagues, suggests that your description of the “scientific community” is far too noble, but this is anecdotal evidence, as is your claim. But do you think that Richard Dawkins is anything other than an ardent promoter and defender of precisely that kind of thing? What else do you call a man who summarily denies the existence of serious scientists who question NDE (when their existence and “seriousness” is well known)? How about Hitchens and Harris? Or the trio that wrote in to the Tennessean? And so on, and so forth. These are the kinds of spokespeople I was referring to in my last reply, and these are the ones Yoram is primarily trying to counteract, I think.

    • Matt Says:

      Hi DD,

      >>
      It appears you’ve done what I’d mentioned people do in these situations: “Yes, science is not certain. But it’s so certain…” This is a fundamental misunderstanding.
      >>

      No, you are putting words in my mouth. I am saying: science is not certain, but good science is strongly supported by multitudinous data. There is not a binary choice between total certitude and complete ignorance. The vast majority of human knowledge falls in the spectrum between the two. Just because proposition X is not perfectly certain, this does not mean that it’s no better than anything else. Science is not certain, but many claims in science are vastly better supported than anything else out there.

      >>
      So everything is “correct till proven incorrect.”
      >>

      No. I am requiring propositions to satisfy a high burden of proof before I place any weight on them. I cannot afford to wait for certitude, I have to go with the best evidence-based methods I have available. But, I strive to make sure that “everything” is subjected to those standards.

      >>
      Your statement, “If you want to convince me of a Torah proposition, you need to make logical and evidence based arguments,” is false, because if we’re using the same terminology, the Torah does not make propositions, it reveals facts.
      >>

      I have a problem with this rock, paper, scissor approach to Torah (Torah is absolute truth therefore it trumps imperfect, evidence based findings). Imagine I were a Jewish orphan, born on an island with no exposure to Judaism. You want to convince me that the Torah reveals truth. You would need to:

      1) Prove that Torah is divine.
      2) Prove that the Torah we have today is the same Torah we got at Sinai, ie it was passed down perfectly.
      3) That our limited human understanding of the text is correct (or that one opinion in Chazal is more correct than another)

      Any arguments you make for these three points -compelling as they may be- are based on human logic, inference, and evidence, which puts you on the same epistemic level as science. You need to use imperfect human knowledge to get me to the point of even accepting the notion that statements of the Torah are absolutely true.

      >>
      But do you think that Richard Dawkins is anything other than an ardent promoter and defender of precisely that kind of thing?
      >>

      I did not mean to say that there weren’t *any* ideologues in science and Dawkins is indeed a firebrand. There are also some very outspoken advocates for non-Darwinian evolutionary mechanisms. For every Dawkins and anti-Dawkins out there, there are hundreds of excellent biologists who just go on and quietly do good science. The people in those fields who I admire and listen to have a very different tone and approach than Dawkins. I have no problem with Yoram trying to take on Dawkins or Hitchens (who was not a scientist) on. But, I have a problem with him using Dawkins as a straw-man representation of the whole community.

  6. DD Says:

    Matt,

    “I am saying: science is not certain, but good science is strongly supported by multitudinous data. There is not a binary choice between total certitude and complete ignorance. The vast majority of human knowledge falls in the spectrum between the two.”

    Ah, but no one actually set that up as the choice. The choice wasn’t “complete ignorance” vs. “total certitude,” it was “educated guess” vs. “absolute truth.” Nowhere in my reply did I even hint to such a dichotomy. What I did say was “we cannot render what we know to be accurate judgments about the mechanics of the world.” We don’t know them to be accurate. We hope they’re accurate. They solve lots of problems. The point is that in comparison to information issuing from a source which is, by definition, the most correct, there’s no contest. They are in two different categories. By discussing them in the same breath, you are comparing incomparable types of information, and so my characterization of your previous reply would have been correct.

    But the truth is that the end of your reply is implicitly based on a correct understanding of the issue, so I’ll spend more time on that.

    “No. I am requiring propositions to satisfy a high burden of proof before I place any weight on them. I cannot afford to wait for certitude, I have to go with the best evidence-based methods I have available. But, I strive to make sure that “everything” is subjected to those standards.”

    Right. That’s pretty much what everyone who thinks about these problems does. That doesn’t actually render scientific results closer to any absoluteness. They are, in fact, philosophically what I called them: correct until proven incorrect.

    “Any arguments you make for these three points -compelling as they may be- are based on human logic, inference, and evidence, which puts you on the same epistemic level as science. You need to use imperfect human knowledge to get me to the point of even accepting the notion that statements of the Torah are absolutely true.”

    Now we’re talking. Yes. Since we are human, and our methods of gathering information are human, we cannot transcend imperfect levels of knowledge. This is a fundamental fact. Any decisions I make, even the most basic decisions about my life such as “the Torah is what Jews claim it is,” are still decisions involving my human reasoning. My reason could be a belief in my tradition, the impossibility of the mesorah being forged wholesale, or the miracles that happen to righteous Jews. Different people will give different answers.

    What happens when two equally strong chains of logic can be used to come to opposite conclusions? This already took place once with the problem of corporeality in the Torah. R. Saadiah Gaon explicitly states in the beginning of the second ma’amar of Emunos VeDeos that since we know with philosophical certainty that God cannot have a body, we must take all instances of seeming corporeality metaphorically. This is the first instance I know of of a traditional Jewish exegete squaring our interpretations with what our most basic logic tells us. (I would actually cite Talmudic examples, but this will take too long, and is subject to too many differences of opinion among the later commentators)

    However, the problem here is in your overgeneralization. There are epistemologically differing levels of knowledge even within our imperfect human constructs. Sensory information reigns supreme, followed by (or maybe on par with) fundamental logical operations (such as the correct syllogisms in the Prior Analytics). These two are not subject to dispute, because they characterize the way that we operate.

    To a very distant third, however, I relegate all assumptions beyond this. One of the assumptions in this last category is that the constants that govern physical laws have been stable over the entire period of the world’s existence. There is absolutely nothing (on a purely logical level) to get me to think that way. Even if one assumes the materialist hypothesis that “what you see is what you get,” there is zero logical indication of the stability of the patterns governing this grand system! Sure, it seems that today’s gravity is pretty much the same as yesterday, and the day before that, but we can never elevate these assumptions to the level of the first two! If, by contrast, we are able by logical necessity to show that we must accept the Torah as what it is claimed to be, the weakest link is at the level of logical operations.

    In the discussions I’ve had with working lab-coat types, the answer I’ve typically gotten to this is “Yeah, but what about Ockham’s Razor?” If you think this is a serious proposition, I can discuss it in my next reply. Or, if you have other ideas, I’m ready to listen.

    “I have no problem with Yoram trying to take on Dawkins or Hitchens (who was not a scientist) on. But, I have a problem with him using Dawkins as a straw-man representation of the whole community.”

    But I really don’t think he does that! Look, I haven’t read all the posts on this blog, but I have read Genesis and Genes, and I don’t remember him once casting such aspersions on scientists as a group. The only thing I remember him saying is that they are subject to paradigmatic prejudices, which he emphasized was true for all groups of people. He was trying to use this to explain why a large number of scientists agreeing to a paradigm that they’re not in the process of scrutinizing or working with does not imply that there’s the kind of ironclad confirmation that people are taught to believe in. Is he correct? It’s definitely possible.

    • Sam Says:

      I like your posts. Do you have a blog or the sort?

    • Matt Says:

      There is a lot to unpack here, so I will respond in two threads. The first one is concerning the whole paradigmatically biased argument. The second will address the epistemological science vs Torah issues as I see them.

      >>>
      “The only thing I remember him saying is that they are subject to paradigmatic prejudices, which he emphasized was true for all groups of people.”
      >>>

      Here are my issues:

      1.) These types of arguments are, at best, informed innuendo. And they cannot e used selectively as wild-cards against those theories you don’t like. That approach is post-hoc. And, the excellent scholarship of many scientists deserves to be met with precise scientific arguments and not handwaving that there might be bias.

      2.)Yoram can claim that the scientific community has “Darwinist” biases and that they defending the prevalent paradigm. But, I or any other scientist can make strong claims that he himself is hopelessly rigid and dogmatic. We can draw parallels between him and the dogmatic minority the loosing end of a paradigm shift, resorting to confounding rhetorical methods to defend indefensible positions. This whole discussion would the revert into a “he said/she said”…and an unproductive one at that. Let’s stick to the science.

      3.) There are differenced between subtle “paradigmatic biases” and straight up dogmatism. I fully concede the possibility of the former. However, I take strong issue with the claim that the community of evolutionary biologists is broadly dogmatic (though I admit there are exceptions). It goes against my own direct experience and it is belied by the fact that (as Yoram even points out) there is rigorous debate among scientists

      4.) When it comes to skepticism and objectivity, it is true that we are all human and imperfect. But some of communities are more objective and more skeptical than others. The whole point of science is that it at least *attempts* to employ methods that separate out or place checks on bias. Scientists come from a wide variety of backgrounds. They also have competing egos. *Deeply* scrutinizing each other’s work (and our own) is what we do for a living and I think we are quite good at it. A professional ball player and an amateur both strike out from time to time. Both are imperfect, but I want the professional on my team if I’m going up against a real pitcher. The same goes for a professional scientist. Most readers of this blog -and I suspect Yoram himself- do not appreciate the level of deep and sophisticated scrutiny that is applied to good science. Most people skim papers for their conclusions, but the hallmark of good science is that most of the time is put into the error analysis. Those of us who chose careers in science, love the give and take. We love playing the role of both advocates and challengers to the paradigm. Sure we are functionally biased by it (to an extent that can only be seen in hindsight) but the bulk of us really passionately and sincerely strive to keep an open mind. Kuhn, for all his meta-discussions on the subject paradigmatic bias, still thinks that the success of science comes in large part because the culture of science is excellent at these sorts of things. In the end of The Structure of Scientific Revolutions, Kuhn makes the point that one of the essential characteristics of a healthy scientific process is that “the power to choose between paradigms [is vested] in the members of a special kind of community”. He speaks of a “professional” community with a strong commitment to shared values, to high degree of education, and serving as “sole possessors of the rules of the game”. I very much doubt that Kuhn sees the scientific community as just another group with a bias.

  7. DD Says:

    Sam,

    Unfortunately, I don’t have anything to say that’s important enough to publish, but I’m glad you enjoy whatever I’ve written. Also, I’d like to take this opportunity to emphasize that whenever I’ve replied to Matt in a way that seemed that I was certain of my positions, it’s not because I actually have everything worked out. I’m very far from any kind of intellectual perfection, so this is just what makes sense in the ol’ brain right now. It’s not always easy to make sure the writing reflects this and is sufficiently to the point.

  8. Matt Says:

    Now your science and Torah comments:

    “In the discussions I’ve had with working lab-coat types”

    I have never worn a lab coat, except as a Halloween costume once :).

    You claim that arguments for Torah operate on the level of “pure logic” while science operates on the level of “assumptions”. This is simply not true. All of the philosophical attempts to prove Judaism that I have ever heard rely on certain assumptions that I often find to be very tenuous. The Kuzari-style arguments for revelation at Sinai make claims based on supposed principles of human behavior and historic observations/inferences that I don’t see as necessarily true. Personally, I tend to find the intellectual arguments for the divine truth of Torah to be weak. My own acceptance of Torah is based on personal experience and connection. I *wish* the arguments in support of Torah met the level of scientific scrutiny…Life would be a lot easier.

    In short, part of the difference between us is that, as much as I want to, I simply don’t see the two systems as being based in “two equally strong chains of logic “. I certainly don’t see the arguments for Judaism as succeeding in establishing “logical necessity”. Some arguments I find to be somewhat *compelling*, but not on the level of a proof.

    >>
    “One of the assumptions in this last category is that the constants that govern physical laws have been stable over the entire period of the world’s existence. There is absolutely nothing (on a purely logical level) to get me to think that way. ”
    >>

    This is a common misunderstanding. One of the strengths of good science is the ability to leverage the consistency of many different types of physical law. The laws of physics, chemistry, etc are interconnected. Thus, if one constant changes, it can have profound implications for everything else. Modern science typically does *not* assume constancy of natural law. Rather, it assumes that natural law simply be logically self-consistent. One can always say, “Let us assume that parameter or condition X is variable. Then we would see consequence Y” . One can then go out and look for evidence of consequence Y. If it is there, we know that X did change over time. If is contradicted by evidence, then we place strong limits on how much X could have carried.

  9. DD Says:

    Matt,

    This may surprise you, but I’m actually going to reply with “ditto” to most of your post. But first I will outline what I take issue with.

    Concerning bias:

    With regards to 1), I would agree with you if I thought Yoram were trying to get everybody to stop listening to all advocates of theory X based on the general problem of bias. I do not think so, and I feel that this has been made pretty clear throughout the threads. He has two objectives: a) Knock the levels of perceived certitude and consensus down closer to where they actually are, so that people don’t feel like the Torah is telling them one thing and the other is literally staring them in the face. b) Suggest a reason for why so many people might agree to this idea. You might have beef with the very attempt to use b), but this is neither innuendo to discredit a specific theory or to discredit the scientific community in general. I don’t know. As for 2), Yoram does actually provide material to substantiate that claim, at least for individuals. Again, while not the best way to deal directly with an idea, it contributes to his objective b). Concerning 3), I again fail to find any traces of such a thing in Yoram’s writings. If you really want to, please provide the relevant quotations. Concerning 4), I agree, and again do not think that Yoram claims that they are just any group. He merely said that they are potentially subject to the same biases as any other group, an idea that does not enter the minds of many laymen confronted with the absolute pronouncements of the NYT and the like.

    Maybe I’m missing some of the rhetoric. You can provide quotations, if you’d like.

    Concerning Torah and science:

    I should have explained at greater length that I do not think the arguments for Torah Judaism operate on the level of pure logic either. I’m sorry for overusing the word “purely” with connotations that are certainly beyond what I meant. What I really wanted to do was separate arguments based on “what our senses perceive” + “what our mind dictates,” as opposed to “this is probably the case.”

    Let’s take the Kuzari-type argument, for example. Now I am personally not a Kuzari-or-bust kind of guy, and I certainly wouldn’t go as far as R. Dovid Gottleib and claim that from hereon [i.e., having heard the argument] it’s a choice between being rational or irrational. But let’s examine its components.

    As far as I understand, this proof as a few components: 1) the fact that, at some point in history, the Jewish people was convinced that its ancestors received a binding Torah; 2) the fact that life-altering (as in, binding for the generations) Divine pronouncements wouldn’t have gone unmentioned in any generation. (I know that I’m changing this a little from the way it’s presented on whatever website it is.)

    Now 1) is a fact of history (not with all the details of halacha, but with the general fact). Take it back, just for arguments’ sake, to the age of the Tannaim (and Josephus). 2) is a logical assumption based on how people would have and did react. I say “would have” because we are basing them at least partially on ourselves, and “did” because we do actually have examples, both from the Jewish literature and from other literature of the time, of people reacting to such claims, even on a much smaller scale. “Did” is stronger than “would have,” of course.

    Is 2) foolproof? Does it raise the discussion to the level of a mathematical proof? I don’t think so. But it’s based an observation (or two) combined with a basic aspect of human reasoning.

    While leaving open the possibility of error, this is the kind of “logical necessity” that one usually requires in making important life decisions. Nothing outside of Greek philosophy. rigorous mathematics, and other closed systems ever reaches levels beyond this kind of “logical necessity.” The truth is that even there we run into serious problems (Hofstadter’s GEB has an excellent discussion of some of them, with references to the literature), but we will leave that for the moment.

    Now consider, as opposed to that, the assumption that C14 has always had the same rate of decay. What you mentioned in your last post is the consistency of natural laws. What about the present consistency of natural laws (or, the particular consistency we see in them) indicates to us that at all points prior, this consistency was maintained in the same way? As we do not know how all of the constants interact, or whether there are self-correcting mechanisms in the entirety of the system, we are not in a good place to assume anything backwards, even on the level of a strong argument. Let me make this clear: lacking any indication to the contrary, there’s no reason not to go with these assumptions as long as they work. But here we do have an indication to the contrary, and one which comes from a stronger base. I, myself, have never sat down to read a six-hundred page book on radiocarbon dating (although I almost did, before my time got sucked up by something else), so I obviously might be missing some other argument. Am I? If you will refer to independent confirmation, I will take that up in the next reply.

    You should know that I actually am also rooted in what would be most accurately included in “personal experience.” But this is not so relevant, and is another topic altogether.

    I once again offer to be in contact beyond the confines of the comments section. I have a feeling that a lot of this would be much clearer in person, if that were possible. I am especially desirous of such contact because I don’t have any outside of this.

    A freilichen.

  10. Dr. Ben Goldacre and the Reproducibility of Research | Torah Explorer Says:

    […] https://torahexplorer.com/2013/02/10/replication-of-experimental-data/. […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: