Slashdot Mirror
Poor Scientific Research Is Disproportionately Rewarded (economist.com)
A new study calculates a low probability that real effects are actually being detected in psychology, neuroscience and medicine research paper -- and then explains why.
Slashdot reader ananyo writes:
The average statistical power of papers culled from 44 reviews published between 1960 and 2011 was about 24%. The authors built an evolutionary computer model to suggest why and show that poor methods that get "results" will inevitably prosper. They also show that replication efforts cannot stop the degradation of the scientific record as long as science continues to reward the volume of a researcher's publications -- rather than their quality.
The article notes that in a 2015 sample of 100 psychological studies, only 36% of the results could actually be reproduced. Yet the researchers conclude that in the Darwin-esque hunt for funding, "top-performing laboratories will always be those who are able to cut corners." And the article's larger argument is until universities stop rewarding bad science, even subsequent attempts to invalidate those bogus results will be "incapable of correcting the situation no matter how rigorously it is pursued."
The article notes that in a 2015 sample of 100 psychological studies, only 36% of the results could actually be reproduced. Yet the researchers conclude that in the Darwin-esque hunt for funding, "top-performing laboratories will always be those who are able to cut corners." And the article's larger argument is until universities stop rewarding bad science, even subsequent attempts to invalidate those bogus results will be "incapable of correcting the situation no matter how rigorously it is pursued."
Is that a list of authors or the title of your paper?
Poorly written Economist articles are Disproportionately Rewarded with attention and discussion.
Couldn't have anything to do with short term outlook by poor management in companies? Instant results under pressure to perform on the bottom line.
The problem is how funding is allocated. It's competitive and funding cycles are short, usually about three years per award. There's not anywhere close to enough money to go around, so a lot of good research ideas aren't funded. Institutions take a lot of money off the top in F&A or indirect costs, which are frequently abused. Then you fund graduate students, who in my experience tend to rush their work at the end and don't produce research anywhere close to the value of what they are paid. I'd be in favor of mostly doing away with peer reviewed publications in favor of granting open access to the outcomes of research. That means, instead of writing a paper and keeping the data private, make the data and the tools used to process it public. Make the research less competitive for funding and make the review process more transparent. Obviously you still have to ensure that the research is legitimate, but peer review does an awful job of that.
But not climate change research it's the exception.
Nothing new to see - if volume is all that counts - then volume is all we get.
In medical research when we are comparing groups it is normal to specify the power/ do a power calculation
power is a measure of the risk of finding a result when none exists (falsely rejecting the null hypothesis)
the null hypothesis is that your two treatments are equal
more here:
http://powerandsamplesize.com/...
Humorous signatures are over-rated.
...good scientific research is difficult to process. Groundbreaking research is difficult to understand.
You've discovered that cheaters gain an advantage.
Seven puppies were harmed during the making of this post.
Try this. Climate is a system and both "el" are seasonal patterns. Global warming however is the increasing trend in global average temperature meaning that there is more heat being absorbed and retained in the system. Anthropogenic global warming is the idea that the increasing retention of heat energy is powered by the greenhouse effect from CO2, which is something Svante Arrhenius demonstrated more than 100 years ago but which was ignored. Analysis now confirms that the acceleration in trend was caused by the industrial revolution freeing previous stored carbon from fossil fuels into the atmosphere where it formed CO2 and is driving a runaway greenhouse effect increasing global average temperatures year over year. Climate is dynamical system so results can't be predicted with high certainty but the local effects of increased heat are easy to observe, and will result in desertification of previously arable land.
This original study is here.
The study presents an accurate description of how research is funded in the US (biomedical sciences in particular). I can't speak in detail about other countries, but the major issues seem to be the same in other developed nations.
The problem is how do you decide which study to fund. You have 100 scientist asking for money but you can fund only 10 of them. So you must come with some criteria that will allow you to decide which studies are worth pursuing and of these which ones have staff that is capable of completing the work they are proposing. National Institutes of Health (NIH) scores grants on five criteria:
This is like relatively objective way to score. Yes, evaluating the significance, environment and particularly the investigators may get a bit subjective. Keep in mind that each application is discussed by a panel of experts, so individual biases tend to get evened out (group biases are reinforced). The downsides wouldn't matter much if the competition to get the funded wasn't not so fierce and the penalty for not getting funded wasn't as bad as it is. And this is where academic institutions with the help of NIH have created really perverse incentives. First, NIH has decided that they will fund any amount of salary for the investigators and on top of that will provide overhead to the institution. The overhead is money that are not directly required for research and are payed to the institution to support management and facilities. The overhead typically equals 50% to 100% of the direct research costs. A standard 5 year R01 grant with modular budget ($250,000 per year) brings income of $125,000 to $250,000 per year to the institution. If you are university you look at that and think of it as a great deal - you don't have to pay the investigators full (or any) salary, NIH will cover that, and then you get payed when they get funded. Now there is the small problem with tenure. You can't just fire a tenured professor because they can't get NIH funds. So you make getting NIH funds requirement for giving tenure. For tenured faculty you put pressure on them to leave: cut their salary (in many cases down to 25% of what it was), and take away lab space and access to research facilities.
In case you don't see where all this is going, here it is how it looks like from the perspective of a "young" scientist. You have just endured 5-7 years of miserably payed PhD training, another 3-7 years of post-doc with higher but still crappy salary. During this time you probably worked 10-12 hours a day often on weekends (those of you that had to time mouse pregnancies by coming to the lab at 1am to look at their asses, I salute you!). Now you have finally reached the holy grail and you have an academic position on which you can actually support a family. Except, there is a catch. You have 5 years to put together a research team on a limited budget, make "significant" discoveries that you publish, and as a result of that obtain external funding. If you don't do that you get kicke
Clearly, peer review is working as intended in academia.
I'd rather the government fund its own labs more and fund academic less.
I have no bearing on whether global warming is true or not. The amount of money be thrown around to show it is or isn't a problem bothers me. If it is true, the solutions just funnel money to those who already have money at the expense of everybody else, and then they say, you didn't give us enough money ad naseum. THAT is what really worries me.
Such as the poor "science" used by AGW nutjobs who have still not provided a single shred of evidence, yet have a large number of gullible people convinced, including those in the government who fund them. It's an easy way to make a living without actually having to do anything.
I have seen quite a bit of it and know of several CS PhDs that are based on bogus results. The tragedy is that people doing their research properly will take significantly longer and have much diminished chances at an academic career. And this effect propagates: First PhD students advance on bogus results, then they become professors on fraud and finally the whole research field is broken.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
The original research describes the problem in the area of psychology, but the problems are the same there that they are in the bio-sciences. There is a positive glut of bio- and psych- science applicants, which is the real cause. Competition is incredibly fierce, which is what you are seeing in the above problems. "Produce papers or we'll show you the door" && "Get funding or we'll show you the door" && "We'll pay you nothing anyways" are really the byproducts of high competition.
In the Computer Science department, pressures from industry keep this behavior from occurring as prevalently. You don't really see this type of behavior in the academic CS department (where grad students make $60K instead of $20K, and professors make $120K instead of $60K). In the CS department, you don't see people as scared of being thrown out, because industry jobs pay $120K if you are even close to worth your salt. As a byproduct, you see people taking more time with their significant discoveries.
Source: I am a CS PhD working on an interdisciplinary team of psychologists/CS people (70/30 split). I have been on both sided of the recruitment process, as a post-PhD employee into industry (120K) or Government (90K), and as a grad student ($40K for school, or a $70K job which pays for school; chose job). I have evaluated business and academic proposals for multiple agencies, but mostly DoD.
Indeed. But I think there is another angle to this that, while in some ways less obviously sloppy, is in some ways worse.
The need to publish at a furious pace might not always result in cutting corners - indeed, a lab that has done the same thing for the last 10 years might well have refined its techniques to a high level. But in order to do this, they need to remain within rigid boundaries, always using essentially the same methods. Every paper becomes a minor variation on the same old theme. It's the only way you can crank out a dozen papers a year.
For example, the overwhelming bulk of animal-based biomedical research uses mice. If you will, for now, let's avoid the debate about the ethics of using animals in research and just concentrate on the science. Now mice have advantages: they are cheap, breed like crazy, easy to take care of, take up little space, the genetic techniques are standardized etc. And yet, as a model of human disease, they are perhaps the worst possible mammalian model. But these researchers don't dare switch: they are tied to the mouse model and cranking out a dozen papers a year, they can't even consider working on another species.
No I don't know how to fix this either, except to note that while everyone needs competition and pressure to work hard and effectively, at some point too much pressure becomes counterproductive. When the smartest and hardest working people around find it increasingly impossible to make it playing by the rules, well, it should not surprise us if they start to focus more on careerism and less on science...
There are a few well-known issues: not publishing negative results, overinflating importance of your new method, and the drive to publish like crazy discourages collaboration to some extent.
In the first issue, most well-known conferences and journals have so many positive results to choose from (partly because of the second problem), that they just don't care about the negative results. Negative results also don't draw a lot of attention in CS, as they'd pertain to someone trying their own random idea. Some people might think the idea is dumb and would never work -- but who is to say? If it works, then it wasn't so dumb -- but then that's a positive result. If it doesn't work, I can have 1,000 other bridges to publish at your journal by next Tuesday... heh. I think it's a hard problem, but not unmanageable. Maybe some new journals dedicated to negative results (there are some already but aren't known for being very rigorous yet) that at least put your negative result through the gauntlet first.
The second issue is a consequence of a researcher not wanting to accept a negative result -- just find some data that works on your new method and act that's the first dataset you tried it on. This may horrify researchers in other fields, but it definitely happens here. What mitigates this problem, at least, is (a) a lot of research code is on GitHub now, and (b) you can tell what new research is useful whether or not people actually use it in the future. That's the good thing about machine learning, if it works, it'll actually be used somewhere. Basically, we have easy reproducibility.
The last issue is if you have a cool new idea, I've found you don't really want to tell anyone about it until (a) you find out it sucks, or (b) you have it accepted as a paper somewhere. THEN you can collaborate with others.