Slashdot Mirror
Does Journal Peer Review Miss Best and Brightest?
sciencehabit writes: A study published today indicates that the scientific peer review system does a reasonable job of predicting the eventual interest in most papers, but it may fail when it comes to identifying really game-changing research. Papers that were accepted outright by one of the three elite journals tended to garner more citations than papers that were rejected and then published elsewhere (abstract). And papers that were rejected went on to receive fewer citations than papers that were approved by an editor. But there is a serious chink in the armor: All 14 of the most highly cited papers in the study were rejected by the three elite journals, and 12 of those were bounced before they could reach peer review. The finding suggests that unconventional research that falls outside the established lines of thought may be more prone to rejection from top journals.
This is yet another statistical study that itself has flaws.
1.What was the selection process for the studies. The phrase "All 14 of the most highly cited papers in the study" implies that there were papers not in the study. Possible selection bias?
2. They do not go into why the 14 papers were cited so much and if any further research or refinement of the papers were done before they were accepted by other journals. Surface analysis of numbers can be manipulated to say anything.
3. They also say that it might be better to not have a review and just publish everything. This just means that everyone who reads the papers has to do the review. That is not practical. There are many papers that should not be published due to shoddy practices or malfeasance. Instead of trowing out the whole system how about looking at why the 14 papers were rejected and modifying the system accordingly.
4. The article does not give access to the study so we can review it. It is behind a paywall.
I would say the finding confirms that unconventional research that falls outside the established lines of thought may be more prone to rejection from top journals.
It's typical human politics and ideology at work. What would you expect from a large group of people, all with vested interests?
“He’s not deformed, he’s just drunk!”
If you have a new or interesting approach forget about getting grant funding, you only get money in the UK if the work has already been proven to be successful.
There's a lot of scientific work that needs doing that requires high levels of scientific training, creativity and motivation but that isn't groundbreaking science. This scientific work can be anything from preparing scientific animations for educational use to working as a genome sequencing technician.
Maybe I'm just too cynical but, from what I've seen, it's essentially impossible to get grant funding for truly innovative and groundbreaking science. But there's a lot of outside-the-box innovative research that could be done with little more than a scientific researcher's time - i.e. doesn't need millions of dollars of scientific equipment.
So I wonder about whether some 50/50 support scheme could work: pay scientists a decent salary (say, $70K/year) to do routine quantifiable scientific work for half the year but then let them do whatever scientific research they want for the other half the year. You could still require them to account for their time - e.g. ran molecular dynamics simulation in morning took walk to think about results in afternoon. But you wouldn't try to micromanage the topic(s) of their research.
In a certain sense, that's the rose tinted stereotype of the old university professor - gives some lectures to justify his salary but is understood to spend much of his time staring into space and pondering the mysteries of the Universe. The problem is that, with the insane excess of science PhDs all competing for faculty positions these days, to compete successfully university professors have to occupy almost all their time with administrative nonsense (e.g. bringing in more grant money) so they're no longer free to pursue the big questions (which would involve way too much failure to be acceptable to a modern university administration).
In a certain sense, the key to promoting groundbreaking science is to reduce the accountability and competitive pressure and create an environment where scientists can muck about at the fringes of human knowledge - looking for really interesting stuff but most of the time failing to find anything at all.
Also, a lot of highly-cited papers are methodological in nature, and the "Big 3" tend not to publish many of those.
I agree with you, but will point out that TFA is discussing something else. Science is not about having a majority understand, or be "interested". This means that real science tends to be ignored. Sometimes real science goes against the grain, and that science is never peer reviewed either. Meaning that the same status quo is maintained even if it's not scientific.
Slashdot's biggest topics are not "Science" but politics. This is an area which is extremely complex and there may not be a definitive answer to a problem.
That said, I can name several topics where one side will be censored by moderation. I can do this with a science topics as well as political topics, and anyone that's been here for a while knows the same thing. Slashdot can do this because it's not a scientific paper publishing site, it's an OP/ED site primarily focused on topics geeks like.
"Science" today is in trouble, don't get me wrong. Watching bogus papers get published and treated as fact has become all too common. Some have been outright spoofs, but others are published for political agendas. The latter is extremely troubling for society..
-The wise argue that there are few absolutes, the fool argues that there are no probabilities.
>While I don't disagree with the conclusions, this summary equates paper "quality" with number of citations.
No, they don't "equate". They provide a metric and this metric seems reasonable. Black or white is not applicable here. We are talking about high probability of high quality. Does not work each time. The metric will sometimes miss some good paper but high cited paper are important papers.
>High numbers of citations do not mean high quality, and is very field-dependent.
>Quality can only be assessed by people reading the paper.
You mean that people who assess won't cite a good paper they read? They won't recommend it? That is a nonsense. High number of citation means major paper. However, low number of citation does not mean low quality.