Greek 'meta-researcher' finds much medical research swiss cheese

David H. Freedman, in the November 2010 edition of The Atlantic, documents some explosive research being conducted at the Univeristy of Ioannina's medical school in Greece. One study shows that, for whatever reason, the apendices removed from patients with Albanian names in six Greek hospitals were more than three times as likely to be perfectly healthy as those removed from patients with Greek names.(P.76)
He interviewed Dr. Jophn Ioannidis, a 'meta-researcher,' who has become one of the world's foremost experts on the credibility of medical research. He and his team have shown, again and again, and in many different ways, that much of what biomedical researchers conclude in published studies--conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain--is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed....
In poring over medical journals, (Ioannidis) was struck by how many findings of all types were refuted by later findings.(p.78)
The studies were biased, (given that) the researchers headed into their studies wanting certain results--and lo and behold they were getting them. (He quotes Ioannidis)
 'There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.' (p.80)
To get funding and tenured positions, and often merely to stay afloat, researchers have to get their work published in well-regarded jounrals, where rejection rates can climb above 90 percent. Not surprisingly, the studies that tend to make the grade are those with eye-catching findings....and attempting to undermine the work of respected colleagues can have ugly professional repersussions.
'A pervasive theme of ancient Greek literature is that you need to pursuse the truth, no matter what the truth might be,' (Ioannidis) says.
In one published paper Ioannidis laid out a detailed mathematical proof that, assuming modest levels of researcher bias, typically imperfect research techniques, and the well-known tendency to focus on exciting rather than highly plausible theories, researchers will come up with wrong findings most of the time. Simply put, if you're attracted to ideas that have a good change of being wrong, and if you're motivated to prove them right, and if you have a little wiggle room in how you assemble the evidence, you'll probably succeed in proving wrong theories right. His model predicted, in different fields of medical research, rates of wrongness roughly corresponding to the observed rates at which findings were later convincingly refuted: 80 percent of non-randomized trials, and as much as 10 percent of the platinum-standard large randomized trials. The article spelled out his belief that researchers were frequently manipulating data analyses, chasing career-advancing findings rather than good science, and even using the peer-review process--in which journals ask researchers to help decide which studies to publish--to suppress opposing views. (p.80)
In another paper Ioannidis looked at forty nine of the most highly regarded research findings in medicine over the last 13 years...Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. (p.81)...
'Often the claims made by studies are so exaggerated that you can immediately cross them out without needing to know much about the specific problems with the studies,' Ioannidis says.
But of course, it's that very extravagance of claim (one large randomized controlled trial even proved that secret prayer by unknown parties can save the lives of heart-surgery patients, while another proved that secret prayer can harm them) that helps get these findings into journals and then into our treatments and lifestyles.(p. 84)
'Even when the evidence shows that a particular research idea is wrong, if you have thousands of scientists who have invested their careers in it, they'll continue to publish papers on it,' he says, 'it's like an epidemic, in the sense that they're infected with these wrong ideas, and they're spreading it to other researhers through journals.'
Another anomaly: even when a research error is outed, it typically persists for years or even decades.
He looked at three prominent health studies from the 1980's and 1990's that were each later soundly refuted and discovered that researchers continued to cite the original results as correcct more often than as flawed--in one case for at last 12 years after the results were discredited.(p.85)
And finally:
We could solve much of the wrongness problem, Ioannidis says, if the world simply stopped expecting scientists to be right.That's because being wrong in science is fine, and even necessary--as long as scientists recognize that they blew it, report their mistake openly instead of disguising  it as a success, adn then move onto the next thing, until they come up with the very occasional genuine breakthrough. But as long as careers remain contingent on producing a stream of research that's dressed up to seem more right than it is, scientists will deep delivering exactly that.
"Science in a noble endeavor, but it's also a low-yield endeavor,' he says. 'I'm not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life. We should be very comfortable with that fact. (p.86)
For the whole piece, see
Brave Thinkers

November 2010 ATLANTIC MAGAZINE 
Lies, Damned Lies, and Medical Science
By David H. Freedman