Why Most Published Research Findings Are False

Hugh Pickens writes "Researchers have found that the winner's curse may apply to the publication of scientific papers and that incorrect findings are more likely to end up in print than correct findings. Dr John Ioannidis bases his argument about incorrect research partly on a study of 49 papers on the effectiveness of medical interventions published in leading journals that had been cited by more than 1,000 other scientists, and his finding that, within only a few years, almost a third of the papers had been refuted by other studies. Ioannidis argues that scientific research is so difficult — the sample sizes must be big and the analysis rigorous — that most research may end up being wrong, and the 'hotter' the field, the greater the competition is, and the more likely that published research in top journals could be wrong. Another study earlier this year found that among the studies submitted to the FDA about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results saw print, although negative results are potentially just as informative as positive (if less exciting)."

Comments

Thin Papers Hide Bad Work

[PingPongBoy]

I really hate reading research papers. Many of them are cookie cutter papers to make it look like work was done - a bunch of text with a few graphs and lists thrown in, all followed by a massive list of references. The descriptions defy anyone to actually replicate the work. Theorems are proved on the basis of some other theorems in difficult-to-find references, but the logical steps (as you would see in a math textbook where theorems are proven in detail) are at best a few scattered mentions of the format "if x and y then z (and you better believe it because that's what happens when you do all the substitutions)."

In the past, people didn't have such a huge reference base so they could follow the logic, but now with computers, the Internet, and massive hard drives, papers ought to be much longer and more detailed. This would force researchers to not have the attitude of "because you are either studying for a Ph.D. or you have one, you should have the IQ to reverse engineer my logic".

When people start their education, they are told "show your work". Full credit is not given for just the final answer. Because there are more and more universities, institutes, students, problems, etc., people have less time to read about each project and follow every step of reasoning. It was necessary to keep papers short, and that wasn't such a bad thing when people took care to present well, but the system of trust can only persist as long as the trust is not broken. There are too many research areas now with their own little symbolisms and patterns of communications, as well as too many researchers who invent their own symbolisms and styles when they are unsure whether any standard exists. It's a Tower of Babel.

A solution in the computer age is quite simple. Computer storage is cheap enough to permit massive appendixes that give the details of derivations. The Internet can be used to distribute standard ways of expressing ideas and themes that are commonly found. The entire system needs to be more self enforcing by having papers widely available so that people can see what is the right way versus the wrong way. Then, the statistic of which papers have the most references can give a meaningful idea of which papers are the best.

When I see movies depicting life decades ago, I see that people presented themselves with greater complexity and attention to detail. Communications seem to be more bursty now, perhaps because everyone is trying to finish quicker with every objective. Often this leads to shallower thought though because there isn't time taken or given to ponder. So we may well be seeing "research" that just marginally advances a randomly selected result from someone else's papers, and that is the easy path to getting credit and "getting on with life". It's the whole attitude of "no one is going to care because there is so much going on and I'm just insignificant".

Computers can help here. If people want to achieve more signficance, they can produce more full-bodied writeups - this process itself forces them to think deeper and better, and if they have something worth telling, the world will find out.

Re:Peer review helps

[ShieldW0lf]

The problem is money. If we lived in a world where you gained power through the trust and esteem of your fellow man, no one would publish things they weren't confident in. But we don't. We live in a world where you gain power through the accumulation of leverage that allows you to dominate your fellow man despite their wishes and opinions. In this world, who gives a shit if you were wrong or not? Who gives a shit if anyone knows? It doesn't matter, as long as you got paid before they find out. Then you can trample over them either way.

You want to fix it, you move the power basis from economics to politics. Then, people will actually start caring about their reputation. Without addressing this fundamental problem, nothing will change in the slightest.

Need a revolution to do that though... the incumbents will without a doubt use any and all forces that their disposal to prevent a fair and equitable system from ever coming into existence.

Re:Peer review helps

[ShieldW0lf]

The evidence lies in the pairing. When you see a political system that has an inherent nature that allows self-determination and involvement, it is paired with an economic system that subverts it. When you see an economic system that does the same, it's paired with a political system that subverts it. In our world, it's Democracy that gives you freedom and Capitalism that subverts your freedom. In eastern countries, it's Communism that gives you freedom and Totalitarianism that subverts your freedom. Free people follow leadership, and are discerning in how they do it. But none of us on this planet live in systems that would allow that to occur.