For many years, I assumed that claims made for scientific studies were valid and fact-based, in other words, real science. I thought there was something pure and respected about science.

However, several researchers, including two recent ones, have exposed that many so-called scientific studies are not actually true science.

When you hear about a new study that has been published in a peer-reviewed scientific journal, do you assume that it is valid and true as I once did? Do you accept at face value the conclusions of the researchers? We see stories everyday about studies where the researchers claim (or at least the media reports claim) to have found a cure for some form of cancer, or a vaccine that will prevent HIV, or a treatment that will save lives.

As these researchers have uncovered, we need to be skeptical and to question the data and conclusions of “science-based” reports, and we need to be cautious about trusting their claims.

Several reviews have tried to determine if published scientific studies are valid, and surprisingly to some, they have not been able to validate all or even most of the published studies.

In the most recent report, Michael Hiltzik, writing for the Business section of the Los Angeles Times, says that “Science has lost its way, at a big cost to humanity.” “Researchers are rewarded for splashy findings, not for double-checking accuracy. So many scientists looking for cures to diseases have been building on ideas that aren’t even true.” ⁽¹⁾

Hiltzik cites two reviews that indicate that our faith in fact-based science is probably not justified. In the first review, scientists at the biotech firm Amgen wanted to verify that the millions of dollars they spent on scientific research was based on valid data. They looked at “53 landmark papers in their fields of cancer research and blood biology.” They were shocked to find that only 6 of the 53 studies could be validated. ⁽¹⁾

Another group at “Bayer HealthCare in Germany similarly found that only 25% of published papers on which it was basing R&D projects could be validated, suggesting that projects in which the firm had sunk huge resources should be abandoned. Whole fields of research, including some in which patients were already participating in clinical trials, are based on science that hasn’t been, and possibly can’t be, validated.” [emphasis mine] ⁽¹⁾

Why does this happen? Michael Eisen, a biologist at UC Berkeley and the Howard Hughes Medical Institute, says:

. . . the drive to land a paper in a top journal — Nature and Science lead the list — encourages researchers to hype their results, especially in the life sciences. Peer review, in which a paper is checked out by eminent scientists before publication, isn’t a safeguard. Eisen says the unpaid reviewers seldom have the time or inclination to examine a study enough to unearth errors or flaws.

The journals want the papers that make the sexiest claims, . . .And scientists believe that the way you succeed is having splashy papers in Science or Nature — it’s not bad for them if a paper turns out to be wrong, if it’s gotten a lot of attention. ⁽¹⁾

John Bohannon found out just how serious is the problem with “peer-reviewed” scientific studies.  In his article Who’s Afraid of Peer Review? he wrote, on Science.com, about an experiment where he submitted bogus scientific papers to numerous open-access scientific journals and, to his surprise, more than half of the 304 bogus submissions were accepted for publication! ⁽²⁾

To ensure that the bogus papers wouldn’t pass a thorough review by science experts, Bohannon included numerous “red flags” in them. He says that anyone with “a high-school knowledge of chemistry and the ability to understand a basic data plot should have spotted the paper’s short-comings immediately.” He calls the data flawed and the experiments “outrageous.” ⁽²⁾ The shortcomings certainly should have been found.

Bohannon did have help from researchers at Harvard to “fine-tune the scientific flaws so that they were both obvious and ‘boringly bad.'” There was no sign of peer-review in 60% of the submitted papers that went through the entire editing process. ⁽²⁾

Previous researchers have found similar problems with “scientific” studies.

Lies, Damned Lies, and Medical Science by David H. Freedman tells about the research of Dr. John Ioannidis, who has “spent his career challenging his peers by exposing their bad science.” ⁽³⁾

John Dudley Miller, PhD, says about research studies:

Most people believe that if a research study is published in a peer-reviewed journal, its conclusions are almost certainly true. If they weren’t, the reviewers–other professionals in the field–would certainly catch any errors.

Unfortunately, that’s false.

One quarter of all research studies published in prominent medical journals contain an error in their research design or statistics so important, that error draws the study’s main conclusions into question. ⁽⁴⁾

Dr. Miller says these questionable studies can appear in highly respected journals “of quality and prestige” such as the New England Journal of Medicine, Journal of the American Medical Association, etc. ⁽⁴⁾

What do these reviews tell us? We shouldn’t automatically believe the conclusions of “studies” that claim to be “science-based”! We should look beyond, around, and under the conclusions and mainstream media headlines to find out what the study is really about and what, if anything, it really proves. The next time we see a headline touting a study that proves something we find hard to believe, we shouldn’t assume it’s true! Look for more information.

We should not allow ourselves to be bullied by people who emphatically and arrogantly claim that the safety of GMOs, fluoride, mercury amalgam, glyphosate (Roundup), etc., is based on science. We shouldn’t be discouraged when people say we are “anti-science” because we believe in the value of good nutrition to health and use alternative, traditional health care practices. We just need to remember that being “science-based” isn’t necessarily proof of scientific validity.

We can improve our ability to assess the validity of scientific studies, and I plan to write more about how to do that in a future post.

Update, November 21, 2013: Medicine’s House of Cards – What Happens When We’ve Got It All Wrong. Dr. Kelly Brogan says that “I understand “now, the role of industry bias in publication of studies, the design limitations of randomized trials in accounting for biochemical individuality, and the many permissible aspects of randomized trials that allow for skewed outcomes (placebo washout, breaking blind with inert placebo, allowance of sedatives, etc). I now understand that health is about so much more than is factored into these trials.”

Sources

¹ Michael Hiltzik, Science has lost its way, at a big cost to humanity
² John Bohannon, Who’s Afraid of Peer Review?
³ David H. Freedman, Lies, Damned Lies, and Medical Science
⁴ John Dudley Miller, PhD, Evaluating Scientific Research

This post is shared on Real Food Wednesday and Fight Back Friday.