Poor Scientific Research Is Disproportionately Rewarded (economist.com)

A new study calculates a low probability that real effects are actually being detected in psychology, neuroscience and medicine research paper -- and then explains why. Slashdot reader ananyo writes: The average statistical power of papers culled from 44 reviews published between 1960 and 2011 was about 24%. The authors built an evolutionary computer model to suggest why and show that poor methods that get "results" will inevitably prosper. They also show that replication efforts cannot stop the degradation of the scientific record as long as science continues to reward the volume of a researcher's publications -- rather than their quality.
The article notes that in a 2015 sample of 100 psychological studies, only 36% of the results could actually be reproduced. Yet the researchers conclude that in the Darwin-esque hunt for funding, "top-performing laboratories will always be those who are able to cut corners." And the article's larger argument is until universities stop rewarding bad science, even subsequent attempts to invalidate those bogus results will be "incapable of correcting the situation no matter how rigorously it is pursued."

Comments

Re: management

[Anonymous Coward]

No, a lot of this work is done in academic institutions that rely on grants to fund research. The funding cycles tend to be about three years long, and there's pressure to generate lots of publications rather than do good work. Institutions also tend to skim a lot of money off the top through F&A costs, and there's a lot of corruption involved. As a result, money isn't spent well and there's not enough to go around.

Re:management

[ShanghaiBill]

Couldn't have anything to do with short term outlook by poor management in companies?

Very few companies do any published research. This is about academia and government funded labs that seek grants, not industry.

The "short term outlook" in companies actually improves the situation, because it puts pressure on researchers to come up with real results that can be put into products, rather than bogus research papers.

Medical research

[Sad Loser]

In medical research when we are comparing groups it is normal to specify the power/ do a power calculation

power is a measure of the risk of finding a result when none exists (falsely rejecting the null hypothesis)

the null hypothesis is that your two treatments are equal

more here:
http://powerandsamplesize.com/...

Re: Medical research

[Cinnamon Beige]

You are also supposed to mention those things with your bog standard psychology papers--and in several places, including the abstract, where you cover your sample size and your alpha as well as what you got as an effect size. This doesn't really do much, though, if the entire system is skewed to encourage generally weak work.

With the neuroscience papers involved in this analysis, though, I would want to know what they're looking at. Some papers oughtn't be counted simply because the research relies on people who have neurological damage and you can't do much if you can't find that many people with that problem--well, okay, a lot can be done with carefully applied violence but it tends to get frowned upon for some reason. So, well, some of these studies are weak simply because there wasn't much else that could be done, and some are done basically to go "No seriously can we dig up more test subects?!" (If the results are sufficiently dramatic, you might manage to get more people involved in the search through haystack so you can find the 'needles' required...or, anyway, more of them.)

Re:Change the funding cycles

[NotAPK]

Are you serious?

"Then you fund graduate students, who in my experience tend to rush their work at the end and don't produce research anywhere close to the value of what they are paid."

Grad students are paid barely above minimum wage, if that. They actually aren't expected to produce *any* research output, and anything they get out of their project is regarded as a bonus. Remember, a PhD is a *training* exercise and students are *learning* how to become scientists, no matter how "good" they may seem. This doesn't stop many grad students being exploited. You'd be hard pressed to find a smarter more "capable" (I put that in scare quotes since some grads can't even tie their shoes) group of people being treated like dirt and generally undervalued. They only tolerate it because they're clueless or they just want to tough it out and get their qualification and move on. For yourself, if you are running your research group on the output of grad students (and yes, I know many are) then you're bound to be sunk sooner or later. Remember: pay peanuts, get monkeys!!

It's a strange claim to make, since hardly anyone in science is overpaid. The discrepancies become apparent once you scale income against level of responsibility, perhaps crudely converted to dollar terms based on the equipment they are using/responsible for. It's not uncommon to find a post-doc managing $2-5 million worth of equipment while being paid $40-60 per year. In the private sector such a management policy would be viewed as fascicle at best and negligent at worst.

I do agree with you entirely on one point: the administrative overheads charged against grants are disgustingly inflated by parasitic policies.

Re: But not climate change research

[ClickOnThis]

The "massive consensus" has been going down every year, more and more scientists are pulling out of the consensus. You will rarly hear about that because politicians and news organizations make a lot of money in making people think it is real.

Citation please?

All of the climate change data sets are made by computer models which always get out the results desired, and the desired result is confirming climate change, because if it does not, their funding is cut. So politicians, news organizations AND scientists benefit from lying, the ones that disprove it are shouted down. And the results? Billions of tax payer money (all of it that our children will have to pay) get sent over to other countries.

You have it backwards. Models are constructed from data, not the other way around. To paraphrase plasma physicist Kenneth Birdsall, the purpose of models is to generate insight, not data.

36%? Yea, there is a reason why I don't believe in any science study unless it makes sense.

Strawman, and a sloppy one at that. The 36% in TFS refers to reproducibility of psychological studies, not climate studies.

This formalizes what is already known

[Anonymous Coward]

This original study is here.

The study presents an accurate description of how research is funded in the US (biomedical sciences in particular). I can't speak in detail about other countries, but the major issues seem to be the same in other developed nations.

The problem is how do you decide which study to fund. You have 100 scientist asking for money but you can fund only 10 of them. So you must come with some criteria that will allow you to decide which studies are worth pursuing and of these which ones have staff that is capable of completing the work they are proposing. National Institutes of Health (NIH) scores grants on five criteria:

1. 1. Significance - if the proposed research pans out how significant will its impact be
2. 2. Innovation - are unexplored areas and ground breaking theories being investigated, are new tools and methods being developed, etc
3. 3. Investigators - if a new investigator is applying, how well has he/she been trained in the past. If an established investigators is applying what matters most is his past contributions (the euphemism being used is "productivity")
4. 4. Approach - the reviewers evaluate how well designed the research approach is. Will it produce the desired results, does it account for all factors that may influence the results, are all necessary controls included, etc.
5. 5. Environment - is all the necessary equipment and facilities available; are there any other factors that may help the research, like intellectual environment, diversity of experts at the site that can be engaged, etc.

This is like relatively objective way to score. Yes, evaluating the significance, environment and particularly the investigators may get a bit subjective. Keep in mind that each application is discussed by a panel of experts, so individual biases tend to get evened out (group biases are reinforced). The downsides wouldn't matter much if the competition to get the funded wasn't not so fierce and the penalty for not getting funded wasn't as bad as it is. And this is where academic institutions with the help of NIH have created really perverse incentives. First, NIH has decided that they will fund any amount of salary for the investigators and on top of that will provide overhead to the institution. The overhead is money that are not directly required for research and are payed to the institution to support management and facilities. The overhead typically equals 50% to 100% of the direct research costs. A standard 5 year R01 grant with modular budget ($250,000 per year) brings income of $125,000 to $250,000 per year to the institution. If you are university you look at that and think of it as a great deal - you don't have to pay the investigators full (or any) salary, NIH will cover that, and then you get payed when they get funded. Now there is the small problem with tenure. You can't just fire a tenured professor because they can't get NIH funds. So you make getting NIH funds requirement for giving tenure. For tenured faculty you put pressure on them to leave: cut their salary (in many cases down to 25% of what it was), and take away lab space and access to research facilities.

In case you don't see where all this is going, here it is how it looks like from the perspective of a "young" scientist. You have just endured 5-7 years of miserably payed PhD training, another 3-7 years of post-doc with higher but still crappy salary. During this time you probably worked 10-12 hours a day often on weekends (those of you that had to time mouse pregnancies by coming to the lab at 1am to look at their asses, I salute you!). Now you have finally reached the holy grail and you have an academic position on which you can actually support a family. Except, there is a catch. You have 5 years to put together a research team on a limited budget, make "significant" discoveries that you publish, and as a result of that obtain external funding. If you don't do that you get kicke

Re: But not climate change research

[PopeRatzo]

If your paper confirms climate change, you are more likely to get funding.

If your paper confirms that GMOs are as safe as mother's milk, you are also more likely to get funding. Also, if your study shows that vaccines are safe, you are more likely to get funding.

Are those examples of confirmation bias too?

Re: But not climate change research

[TapeCutter]

If your paper confirms climate change, you are more likely to get funding.

If your paper disproves it you get a Nobel prize.

Also happens in CS research

[gweihir]

I have seen quite a bit of it and know of several CS PhDs that are based on bogus results. The tragedy is that people doing their research properly will take significantly longer and have much diminished chances at an academic career. And this effect propagates: First PhD students advance on bogus results, then they become professors on fraud and finally the whole research field is broken.

Re:But not climate change research

[gweihir]

Please be self-destructive _without_ dragging the rest of the human race into it. While there surely is some bogus climate-change research, the whole field is not broken and the whole field has a consensus that it is going to be at least pretty bad and may well get catastrophic.

Re:Change the funding cycles

[umafuckit]

Grad students are paid barely above minimum wage, if that. They actually aren't expected to produce *any* research output, and anything they get out of their project is regarded as a bonus.

I don't know what field you're coming from, but that's not the case in neuroscience. Anyone coming out of a PhD in this field with no publications isn't going to be happy with their performance and it will likely count against them in looking for a good Post Doc.

Re:Change the funding cycles

[serviscope_minor]

You're not quite on the mark there. PhD students are indeed learning how to become research scientists, and the way they practice and prove they have learned is by doing original research. A thesis has to have original research in it or it's not a thesis. In almost all cases that is published somewhere peer reviewed as well.

Re:Change the funding cycles

[serviscope_minor]

There are also "taught" graduate degrees opposed to research degrees.

Yes, but you're talking about graduate degrees, not PhDs. A PhD is a research course. Some have a taught component, sometimes even a whole year, but that's to bring the student up to speed, and so is simply pass/fail with no further effect after a pass. I've examined/viva'd a couple of PhDs, and original research was a major part of the criteria for examination.