Back in 2005, John Ioannidis, who is now a professor of medicine at Stanford, made headlines by announcing that half the results published in peer-reviewed scientific journals were probably wrong. And since then, “researchers have confirmed his skepticism by trying—and often failing—to reproduce many influential journal articles.”1But, what about our government? Are they concerned about the science behind the policies they are making?

RELATED STORY:

The biggest issues have come from psychology studies: in a 2015 article in Science, independent researchers tried to replicate 100 prominent psychology studies and only succeeded in 39% of them. And in 2012 the biotechnology firm Amgen tried to reproduce 53 “landmark” studies in hematology and oncology but could only replicate six. And these are only two examples, the problem is far more serious.

However, it’s not just that doctors are basing medical decisions on incorrect information, it’s about the financial costs, too; “A 2015 study estimated that American researchers spend $28 billion a year on irreproducible preclinical research.”2 For that much money, shouldn’t researchers be getting things right more than wrong?

Well, it’s possible that the main cause of irreproducibility is that scientists are looking for stuff that just isn’t there, either on purpose or accidentally; if a researcher looks long enough, they can turn “any fluke correlation into a seemingly positive result.” And they are susceptible to groupthink, aren’t necessarily as skeptical of results that fit their biases, and negative results usually disappear because new findings offer notoriety and job stability. Besides, there is little reward for replication studies and certainly not enough money in it.

Thankfully, American science has begun to face up to these problems. The National Institutes of Health has strengthened its reproducibility standards and scientific journals have reduced the incentives and opportunities to publish bad research. But more needs to be done and the National Association of Scholars has made some recommendations:3
  • Before conducting a study, scientists should “preregister” their research protocols by posting the intended methodology online, which eliminates opportunities for changing the rules in the middle of the experiment.
  • High schools, colleges, and graduate schools need to improve science education, particularly in statistics.
  • Universities and journals should create incentives for researchers to publish negative results.
  • Scientific associations should seek to disrupt disciplinary groupthink by putting their favored ideas up for review by experts in other sciences.

This irreproducibility crisis has largely been kept from the general public and policymakers. But, given that the government relies on these “supposed” scientific findings to make its decisions its time for a change. Federal agencies could help that process along by immediately adopting the NIH’s new standards and Congress could pass a law to ensure that all future regulations are based on those same high standards.

“Each scientific discipline needs to accept responsibility for its share of the irreproducibility crisis and incorporate strict standards into its procedures. The goal must be to reinvigorate the tradition of scientific inquiry. What the crisis teaches is that the scientific spirit lies with those who constantly test for that fundamental requirement of truth—that a result can be reproduced.”4

The question is will they? Or are they currently getting exactly what they want?

Sources and References

  1. WSJ, April 16, 2018.
  2. WSJ, April 16, 2018.
  3. WSJ, April 16, 2018.
  4. WSJ, April 16, 2018.