Slashdot Mirror
Poor Scientific Research Is Disproportionately Rewarded (economist.com)
A new study calculates a low probability that real effects are actually being detected in psychology, neuroscience and medicine research paper -- and then explains why.
Slashdot reader ananyo writes:
The average statistical power of papers culled from 44 reviews published between 1960 and 2011 was about 24%. The authors built an evolutionary computer model to suggest why and show that poor methods that get "results" will inevitably prosper. They also show that replication efforts cannot stop the degradation of the scientific record as long as science continues to reward the volume of a researcher's publications -- rather than their quality.
The article notes that in a 2015 sample of 100 psychological studies, only 36% of the results could actually be reproduced. Yet the researchers conclude that in the Darwin-esque hunt for funding, "top-performing laboratories will always be those who are able to cut corners." And the article's larger argument is until universities stop rewarding bad science, even subsequent attempts to invalidate those bogus results will be "incapable of correcting the situation no matter how rigorously it is pursued."
The article notes that in a 2015 sample of 100 psychological studies, only 36% of the results could actually be reproduced. Yet the researchers conclude that in the Darwin-esque hunt for funding, "top-performing laboratories will always be those who are able to cut corners." And the article's larger argument is until universities stop rewarding bad science, even subsequent attempts to invalidate those bogus results will be "incapable of correcting the situation no matter how rigorously it is pursued."
Poorly written Economist articles are Disproportionately Rewarded with attention and discussion.
Couldn't have anything to do with short term outlook by poor management in companies? Instant results under pressure to perform on the bottom line.
In medical research when we are comparing groups it is normal to specify the power/ do a power calculation
power is a measure of the risk of finding a result when none exists (falsely rejecting the null hypothesis)
the null hypothesis is that your two treatments are equal
more here:
http://powerandsamplesize.com/...
Humorous signatures are over-rated.
Are you serious?
"Then you fund graduate students, who in my experience tend to rush their work at the end and don't produce research anywhere close to the value of what they are paid."
Grad students are paid barely above minimum wage, if that. They actually aren't expected to produce *any* research output, and anything they get out of their project is regarded as a bonus. Remember, a PhD is a *training* exercise and students are *learning* how to become scientists, no matter how "good" they may seem. This doesn't stop many grad students being exploited. You'd be hard pressed to find a smarter more "capable" (I put that in scare quotes since some grads can't even tie their shoes) group of people being treated like dirt and generally undervalued. They only tolerate it because they're clueless or they just want to tough it out and get their qualification and move on. For yourself, if you are running your research group on the output of grad students (and yes, I know many are) then you're bound to be sunk sooner or later. Remember: pay peanuts, get monkeys!!
It's a strange claim to make, since hardly anyone in science is overpaid. The discrepancies become apparent once you scale income against level of responsibility, perhaps crudely converted to dollar terms based on the equipment they are using/responsible for. It's not uncommon to find a post-doc managing $2-5 million worth of equipment while being paid $40-60 per year. In the private sector such a management policy would be viewed as fascicle at best and negligent at worst.
I do agree with you entirely on one point: the administrative overheads charged against grants are disgustingly inflated by parasitic policies.
*farcical
The "massive consensus" has been going down every year, more and more scientists are pulling out of the consensus. You will rarly hear about that because politicians and news organizations make a lot of money in making people think it is real.
Citation please?
All of the climate change data sets are made by computer models which always get out the results desired, and the desired result is confirming climate change, because if it does not, their funding is cut. So politicians, news organizations AND scientists benefit from lying, the ones that disprove it are shouted down. And the results? Billions of tax payer money (all of it that our children will have to pay) get sent over to other countries.
You have it backwards. Models are constructed from data, not the other way around. To paraphrase plasma physicist Kenneth Birdsall, the purpose of models is to generate insight, not data.
36%? Yea, there is a reason why I don't believe in any science study unless it makes sense.
Strawman, and a sloppy one at that. The 36% in TFS refers to reproducibility of psychological studies, not climate studies.
If it weren't for deadlines, nothing would be late.
You've discovered that cheaters gain an advantage.
Seven puppies were harmed during the making of this post.
Try this. Climate is a system and both "el" are seasonal patterns. Global warming however is the increasing trend in global average temperature meaning that there is more heat being absorbed and retained in the system. Anthropogenic global warming is the idea that the increasing retention of heat energy is powered by the greenhouse effect from CO2, which is something Svante Arrhenius demonstrated more than 100 years ago but which was ignored. Analysis now confirms that the acceleration in trend was caused by the industrial revolution freeing previous stored carbon from fossil fuels into the atmosphere where it formed CO2 and is driving a runaway greenhouse effect increasing global average temperatures year over year. Climate is dynamical system so results can't be predicted with high certainty but the local effects of increased heat are easy to observe, and will result in desertification of previously arable land.
Nein. Consider how many scales are "validated" by Psychology post-docs, either with PCA that they call factor analysis or confirmatory factor analysis. In both cases the measurement structures are wrong (non-continuous data --> a Hessian that's overly inflated, and therefore presumed to be more informative that reality holds): PCA only works for formative measurement structures (i.e., the items cause the latent structure, like the Hollingshead SES scale), whereas nearly all social science measurement assumes a reflexive structure (underlying cause which is reflected by the common answering patterns across items), and can be easily attested to if one reports the Cronbach's alpha measure (a ratio of the signal variance to the total variance: although note that variance is undefined for ordinal variables, from a measurement, not predictive, viewpoint) in the paper. Notice the right wing authoritarianism scale, which claims to be tri-dimensional, with strong Cronbach's alpha reliability, also uses the dimensionality drawn from PCA, on ordinal items, which are mutually exclusive claims, but have never been critiqued. Nearly all scales are drawn up in this fashion. The mathematical test of confirmatory factor analysis actually imposes the reflexive structure, and allows for one to associate specific items with the assumed latent causes: however, the test assumes a multivariate normal joint distribution across the observed information, which is almost never seen in Psychology. This is the reason why the likelihood-ratio test, whose null hypothesis is that the model as specified by the researcher is a good approximation to the data, is nearly universally ignored despite being almost universally found to be significant in applications (look at the kvetching of Les Hayduk about this issue). Part of the issue is the assumption that the items are continuous, which again inflates the information in each item, and misrepresents the nature of the data. Item response theory (or more specifically, multidimensional item response theory) perfectly solves these issues, but it is considered a niche tool for us psychometricians, who can be safely ignored for giving a fuck about what the data says. See the responses of Susan Fiske, an endowed chair Princeton PhD of psychology, as related by Gelman [0].
[0]. http://andrewgelman.com/2016/0...
The Spanish Inquisition of Psychometrics; Burning all the heretics.
You need to be quiet and go sit in a corner reading reliable books until you learn something. Observational studies are legitimate and can demonstrate causal effects with careful sampling to obtain data for analysis and thorough testing of theoretical assumptions from influences.
This original study is here.
The study presents an accurate description of how research is funded in the US (biomedical sciences in particular). I can't speak in detail about other countries, but the major issues seem to be the same in other developed nations.
The problem is how do you decide which study to fund. You have 100 scientist asking for money but you can fund only 10 of them. So you must come with some criteria that will allow you to decide which studies are worth pursuing and of these which ones have staff that is capable of completing the work they are proposing. National Institutes of Health (NIH) scores grants on five criteria:
This is like relatively objective way to score. Yes, evaluating the significance, environment and particularly the investigators may get a bit subjective. Keep in mind that each application is discussed by a panel of experts, so individual biases tend to get evened out (group biases are reinforced). The downsides wouldn't matter much if the competition to get the funded wasn't not so fierce and the penalty for not getting funded wasn't as bad as it is. And this is where academic institutions with the help of NIH have created really perverse incentives. First, NIH has decided that they will fund any amount of salary for the investigators and on top of that will provide overhead to the institution. The overhead is money that are not directly required for research and are payed to the institution to support management and facilities. The overhead typically equals 50% to 100% of the direct research costs. A standard 5 year R01 grant with modular budget ($250,000 per year) brings income of $125,000 to $250,000 per year to the institution. If you are university you look at that and think of it as a great deal - you don't have to pay the investigators full (or any) salary, NIH will cover that, and then you get payed when they get funded. Now there is the small problem with tenure. You can't just fire a tenured professor because they can't get NIH funds. So you make getting NIH funds requirement for giving tenure. For tenured faculty you put pressure on them to leave: cut their salary (in many cases down to 25% of what it was), and take away lab space and access to research facilities.
In case you don't see where all this is going, here it is how it looks like from the perspective of a "young" scientist. You have just endured 5-7 years of miserably payed PhD training, another 3-7 years of post-doc with higher but still crappy salary. During this time you probably worked 10-12 hours a day often on weekends (those of you that had to time mouse pregnancies by coming to the lab at 1am to look at their asses, I salute you!). Now you have finally reached the holy grail and you have an academic position on which you can actually support a family. Except, there is a catch. You have 5 years to put together a research team on a limited budget, make "significant" discoveries that you publish, and as a result of that obtain external funding. If you don't do that you get kicke
Clearly, peer review is working as intended in academia.
I'd rather the government fund its own labs more and fund academic less.
Other exceptions to this rule are studies that show GMOs are safe. Those scientists are impeccable, their studies well-designed and their research should never be questioned. Climate scientists, however, are the bunk.
You are welcome on my lawn.
If your paper confirms that GMOs are as safe as mother's milk, you are also more likely to get funding. Also, if your study shows that vaccines are safe, you are more likely to get funding.
Are those examples of confirmation bias too?
You are welcome on my lawn.
The fact that a computer program produces the same results when executed again may be science, but it's computer science, not climate science.
The reproducability of a computer program proves jack shit about the environment.
If your paper confirms climate change, you are more likely to get funding.
If your paper disproves it you get a Nobel prize.
And did you exchange a walk on part in the war for a lead role in a cage? - Pink Floyd.
I have seen quite a bit of it and know of several CS PhDs that are based on bogus results. The tragedy is that people doing their research properly will take significantly longer and have much diminished chances at an academic career. And this effect propagates: First PhD students advance on bogus results, then they become professors on fraud and finally the whole research field is broken.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Please be self-destructive _without_ dragging the rest of the human race into it. While there surely is some bogus climate-change research, the whole field is not broken and the whole field has a consensus that it is going to be at least pretty bad and may well get catastrophic.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
*$40-60k
Yeah, you're right, that's the second correction to my post. There are probably many others to be made as well...
What would a McDonalds store manager make these days?
How the hell do you reproduce a climate study anyway? Where are the controls?
John Christy has been doing that for over 30 years. For example, he wanted to know if the temperature record was accurate or not. So he developed a secondary way to measure temperature (with satellites). That is one example for you to get the feel, he's done the same thing in other areas of climate science.
"First they came for the slanderers and i said nothing."
Not Kenneth ... In the literature, he was typically listed as CK Birdsall ... for Charles Kennedy ... went by Ned.
(Quick correction from a former phd student of his.)
Oops. Thanks for the correction. I have his plasma physics simulation book, but obviously haven't looked at it recently. :-|
If it weren't for deadlines, nothing would be late.
Grad students are paid barely above minimum wage, if that. They actually aren't expected to produce *any* research output, and anything they get out of their project is regarded as a bonus.
I don't know what field you're coming from, but that's not the case in neuroscience. Anyone coming out of a PhD in this field with no publications isn't going to be happy with their performance and it will likely count against them in looking for a good Post Doc.
soylentnews.org
You're not quite on the mark there. PhD students are indeed learning how to become research scientists, and the way they practice and prove they have learned is by doing original research. A thesis has to have original research in it or it's not a thesis. In almost all cases that is published somewhere peer reviewed as well.
SJW n. One who posts facts.
The original research describes the problem in the area of psychology, but the problems are the same there that they are in the bio-sciences. There is a positive glut of bio- and psych- science applicants, which is the real cause. Competition is incredibly fierce, which is what you are seeing in the above problems. "Produce papers or we'll show you the door" && "Get funding or we'll show you the door" && "We'll pay you nothing anyways" are really the byproducts of high competition.
In the Computer Science department, pressures from industry keep this behavior from occurring as prevalently. You don't really see this type of behavior in the academic CS department (where grad students make $60K instead of $20K, and professors make $120K instead of $60K). In the CS department, you don't see people as scared of being thrown out, because industry jobs pay $120K if you are even close to worth your salt. As a byproduct, you see people taking more time with their significant discoveries.
Source: I am a CS PhD working on an interdisciplinary team of psychologists/CS people (70/30 split). I have been on both sided of the recruitment process, as a post-PhD employee into industry (120K) or Government (90K), and as a grad student ($40K for school, or a $70K job which pays for school; chose job). I have evaluated business and academic proposals for multiple agencies, but mostly DoD.
Here's one answer. (Looks like it tops out at around 55K/year.)
Il n'y a pas de Planet B.
</thread>
(Already posted in this discussion else I'd mod you up.)
Il n'y a pas de Planet B.
I see the sarcasm, but I think you're playing with fire doing it like that.
(I'll STFU now and continue to enjoy your posts quietly. Cheers.)
Il n'y a pas de Planet B.
There is no such consensus.
Please prove your assertion that " has a consensus that it is going to be at least pretty bad and may well get catastrophic".
You see, you lie or repeat lies without even knowing it, which makes it much worse, because thats called ignorance.
Its people who argument/debate like you do, without knowledge who are the most dangerous.
There is such a consensus and it is a strong one. It may need an actual scientist to see it though. You obviously do not qualify.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
So you are basically saying. You have nothing to prove your assertion.
Because I do not believe your blind assertion, I do not qualify?
The supposed consensus is about the earth warming and it "possibly" being caused by humans. And even that consensus is shaky at best.
There i NO concensus at all, about your assertion of thing being really bad or catastrophic, except in the media and young brainwashed eco greenies.
So again, show me the consensus that proves your assertion. Your word, does not count.
There are also "taught" graduate degrees opposed to research degrees.
Yes, but you're talking about graduate degrees, not PhDs. A PhD is a research course. Some have a taught component, sometimes even a whole year, but that's to bring the student up to speed, and so is simply pass/fail with no further effect after a pass. I've examined/viva'd a couple of PhDs, and original research was a major part of the criteria for examination.
SJW n. One who posts facts.
Indeed. But I think there is another angle to this that, while in some ways less obviously sloppy, is in some ways worse.
The need to publish at a furious pace might not always result in cutting corners - indeed, a lab that has done the same thing for the last 10 years might well have refined its techniques to a high level. But in order to do this, they need to remain within rigid boundaries, always using essentially the same methods. Every paper becomes a minor variation on the same old theme. It's the only way you can crank out a dozen papers a year.
For example, the overwhelming bulk of animal-based biomedical research uses mice. If you will, for now, let's avoid the debate about the ethics of using animals in research and just concentrate on the science. Now mice have advantages: they are cheap, breed like crazy, easy to take care of, take up little space, the genetic techniques are standardized etc. And yet, as a model of human disease, they are perhaps the worst possible mammalian model. But these researchers don't dare switch: they are tied to the mouse model and cranking out a dozen papers a year, they can't even consider working on another species.
No I don't know how to fix this either, except to note that while everyone needs competition and pressure to work hard and effectively, at some point too much pressure becomes counterproductive. When the smartest and hardest working people around find it increasingly impossible to make it playing by the rules, well, it should not surprise us if they start to focus more on careerism and less on science...
There are a few well-known issues: not publishing negative results, overinflating importance of your new method, and the drive to publish like crazy discourages collaboration to some extent.
In the first issue, most well-known conferences and journals have so many positive results to choose from (partly because of the second problem), that they just don't care about the negative results. Negative results also don't draw a lot of attention in CS, as they'd pertain to someone trying their own random idea. Some people might think the idea is dumb and would never work -- but who is to say? If it works, then it wasn't so dumb -- but then that's a positive result. If it doesn't work, I can have 1,000 other bridges to publish at your journal by next Tuesday... heh. I think it's a hard problem, but not unmanageable. Maybe some new journals dedicated to negative results (there are some already but aren't known for being very rigorous yet) that at least put your negative result through the gauntlet first.
The second issue is a consequence of a researcher not wanting to accept a negative result -- just find some data that works on your new method and act that's the first dataset you tried it on. This may horrify researchers in other fields, but it definitely happens here. What mitigates this problem, at least, is (a) a lot of research code is on GitHub now, and (b) you can tell what new research is useful whether or not people actually use it in the future. That's the good thing about machine learning, if it works, it'll actually be used somewhere. Basically, we have easy reproducibility.
The last issue is if you have a cool new idea, I've found you don't really want to tell anyone about it until (a) you find out it sucks, or (b) you have it accepted as a paper somewhere. THEN you can collaborate with others.
Disagree. Sort of.
A couple points.
First academic science isn't alone. IT folks have managed and are responsible for systems/databases/applications worth in the tens of millions for similar pay. Probably both in public and private sector.
Second, much of this has as much to due with the glut of Phd's, those hoping to go into academia, and the actual amount of that kind of work available. Much of the glut is probably due to a number of trends, everyone going to university for one (and not all University studies really have practical job applications, particularly those mentioned above). There is room for only so many Theoretical Physicists, or if you look at psychology or sociology only so many therapists. That means the rest are all trying to work at a university someplace doing research and teaching. The requirements (i.e. number of published papers) are high as is the competition. So it shouldn't really be all that surprising that the issue exists. It is a problem of their own making.
I've known some brilliant Phd folks, and I've worked with a bunch that as you say, couldn't tie their shoelaces (even in their field of study). Some I seriously wonder how in this day in age they could ever have passed (simply by their complete lack of computer knowledge, and these are not folks that graduated in the 70's or 80's). Again, think of the issues of the university student mill, and likely professors under pressure to teach, to pass, etc... students thrown at them like a storm.
Anyway who's actual fault is it, and how is it solved? Well that's rather complicated and I won't hazard a guess other than to say it is likely not all that easy with probably a lot of factors largely out of anyone's direct control.