Yanlev/Dreamstime
Experimental replication is a hallmark of the scientific method. The idea is that scientific findings can be considered accurate if any researcher using the same procedures gets the same results. However, that is not what is happening today, as I reported in my article, "Broken Science." For example, in a 2012 study in Nature researchers replicated the findings of only six out of 53 (11 percent) landmark published preclinical cancer studies. In 2011, researchers at Bayer Healthcare reported that they could not replicate 43 of the 67 published preclinical studies that the company had been relying on to develop cancer and cardiovascular treatments and diagnostics.
To address this problem, the Center for Open Science launched the Reproducibilty Project to assess the replicability of research in various disciplines. The first effort focused on trying to reproduce the results of 100 psychology studies. As reported in Science in 2015, just 39 of the studies were successfully replicated. The Open Science Center turned its attention to cancer biology research. The first results of attempts to replicate the findings in five different cancer studies are out, and they are complicated. The researchers were able to essentially reproduce the findings in two studies. The results in two others were not interpretable due to technical problems, and one failed replication. Interestingly, other labs report being able to replicate the results in the study whose results could not be reproduced by researchers working with the Open Science Center.
In trying to reproduce the findings of another one of the studies that reported identifying mutations that boosted the proliferation of melanoma cells, the replicating researchers actually found that the control mice without the mutated cancer cells died faster than the ones with the mutated melanoma. They speculate that the different results might hinge on changes in cell culture or further unidentified mutations in the cancer cells used in the replication experiment. And, of course, it may be that the replication studies are flawed rather than that the original ones are. Only more studies would enable researchers to figure out which is which.
"Science, the pride of modernity, our one source of objective knowledge, is in deep trouble," writes Daniel Sarewitz, a professor at Arizona State University's School for Future Innovation and Society, in his 2016 essay "Saving Science" in The New Atlantis. The pervasive lack of robust reproducibility is confirming Sarewitz' conclusion.
Comments