Poor Scientific Research Is Disproportionately Rewarded (economist.com)

A new study calculates a low probability that real effects are actually being detected in psychology, neuroscience and medicine research paper -- and then explains why. Slashdot reader ananyo writes: The average statistical power of papers culled from 44 reviews published between 1960 and 2011 was about 24%. The authors built an evolutionary computer model to suggest why and show that poor methods that get "results" will inevitably prosper. They also show that replication efforts cannot stop the degradation of the scientific record as long as science continues to reward the volume of a researcher's publications -- rather than their quality.
The article notes that in a 2015 sample of 100 psychological studies, only 36% of the results could actually be reproduced. Yet the researchers conclude that in the Darwin-esque hunt for funding, "top-performing laboratories will always be those who are able to cut corners." And the article's larger argument is until universities stop rewarding bad science, even subsequent attempts to invalidate those bogus results will be "incapable of correcting the situation no matter how rigorously it is pursued."

Comments

Re: management

[Anonymous Coward]

No, a lot of this work is done in academic institutions that rely on grants to fund research. The funding cycles tend to be about three years long, and there's pressure to generate lots of publications rather than do good work. Institutions also tend to skim a lot of money off the top through F&A costs, and there's a lot of corruption involved. As a result, money isn't spent well and there's not enough to go around.

Re:Change the funding cycles

[NotAPK]

Are you serious?

"Then you fund graduate students, who in my experience tend to rush their work at the end and don't produce research anywhere close to the value of what they are paid."

Grad students are paid barely above minimum wage, if that. They actually aren't expected to produce *any* research output, and anything they get out of their project is regarded as a bonus. Remember, a PhD is a *training* exercise and students are *learning* how to become scientists, no matter how "good" they may seem. This doesn't stop many grad students being exploited. You'd be hard pressed to find a smarter more "capable" (I put that in scare quotes since some grads can't even tie their shoes) group of people being treated like dirt and generally undervalued. They only tolerate it because they're clueless or they just want to tough it out and get their qualification and move on. For yourself, if you are running your research group on the output of grad students (and yes, I know many are) then you're bound to be sunk sooner or later. Remember: pay peanuts, get monkeys!!

It's a strange claim to make, since hardly anyone in science is overpaid. The discrepancies become apparent once you scale income against level of responsibility, perhaps crudely converted to dollar terms based on the equipment they are using/responsible for. It's not uncommon to find a post-doc managing $2-5 million worth of equipment while being paid $40-60 per year. In the private sector such a management policy would be viewed as fascicle at best and negligent at worst.

I do agree with you entirely on one point: the administrative overheads charged against grants are disgustingly inflated by parasitic policies.

This formalizes what is already known

[Anonymous Coward]

This original study is here.

The study presents an accurate description of how research is funded in the US (biomedical sciences in particular). I can't speak in detail about other countries, but the major issues seem to be the same in other developed nations.

The problem is how do you decide which study to fund. You have 100 scientist asking for money but you can fund only 10 of them. So you must come with some criteria that will allow you to decide which studies are worth pursuing and of these which ones have staff that is capable of completing the work they are proposing. National Institutes of Health (NIH) scores grants on five criteria:

1. 1. Significance - if the proposed research pans out how significant will its impact be
2. 2. Innovation - are unexplored areas and ground breaking theories being investigated, are new tools and methods being developed, etc
3. 3. Investigators - if a new investigator is applying, how well has he/she been trained in the past. If an established investigators is applying what matters most is his past contributions (the euphemism being used is "productivity")
4. 4. Approach - the reviewers evaluate how well designed the research approach is. Will it produce the desired results, does it account for all factors that may influence the results, are all necessary controls included, etc.
5. 5. Environment - is all the necessary equipment and facilities available; are there any other factors that may help the research, like intellectual environment, diversity of experts at the site that can be engaged, etc.

This is like relatively objective way to score. Yes, evaluating the significance, environment and particularly the investigators may get a bit subjective. Keep in mind that each application is discussed by a panel of experts, so individual biases tend to get evened out (group biases are reinforced). The downsides wouldn't matter much if the competition to get the funded wasn't not so fierce and the penalty for not getting funded wasn't as bad as it is. And this is where academic institutions with the help of NIH have created really perverse incentives. First, NIH has decided that they will fund any amount of salary for the investigators and on top of that will provide overhead to the institution. The overhead is money that are not directly required for research and are payed to the institution to support management and facilities. The overhead typically equals 50% to 100% of the direct research costs. A standard 5 year R01 grant with modular budget ($250,000 per year) brings income of $125,000 to $250,000 per year to the institution. If you are university you look at that and think of it as a great deal - you don't have to pay the investigators full (or any) salary, NIH will cover that, and then you get payed when they get funded. Now there is the small problem with tenure. You can't just fire a tenured professor because they can't get NIH funds. So you make getting NIH funds requirement for giving tenure. For tenured faculty you put pressure on them to leave: cut their salary (in many cases down to 25% of what it was), and take away lab space and access to research facilities.

In case you don't see where all this is going, here it is how it looks like from the perspective of a "young" scientist. You have just endured 5-7 years of miserably payed PhD training, another 3-7 years of post-doc with higher but still crappy salary. During this time you probably worked 10-12 hours a day often on weekends (those of you that had to time mouse pregnancies by coming to the lab at 1am to look at their asses, I salute you!). Now you have finally reached the holy grail and you have an academic position on which you can actually support a family. Except, there is a catch. You have 5 years to put together a research team on a limited budget, make "significant" discoveries that you publish, and as a result of that obtain external funding. If you don't do that you get kicke

Also happens in CS research

[gweihir]

I have seen quite a bit of it and know of several CS PhDs that are based on bogus results. The tragedy is that people doing their research properly will take significantly longer and have much diminished chances at an academic career. And this effect propagates: First PhD students advance on bogus results, then they become professors on fraud and finally the whole research field is broken.