Slashdot Mirror
Does Journal Peer Review Miss Best and Brightest?
sciencehabit writes: A study published today indicates that the scientific peer review system does a reasonable job of predicting the eventual interest in most papers, but it may fail when it comes to identifying really game-changing research. Papers that were accepted outright by one of the three elite journals tended to garner more citations than papers that were rejected and then published elsewhere (abstract). And papers that were rejected went on to receive fewer citations than papers that were approved by an editor. But there is a serious chink in the armor: All 14 of the most highly cited papers in the study were rejected by the three elite journals, and 12 of those were bounced before they could reach peer review. The finding suggests that unconventional research that falls outside the established lines of thought may be more prone to rejection from top journals.
For those of you just joining us, peers given mod points hand them out to review comments posted here. CmdrTaco's site has seen a lot of controversies with this system in the earlier days, and M2 was invented to review the mod point decisions. Lots of discussion has been sorted by this system, and the crap found on other servers has been eliminated.
We cannot solve our problems with the same thinking we used when we created them.
-- Albert Einstein
Full disclosure: I found this at this web site.
-- I ignore anonymous replies to my comments and postings.
While I don't disagree with the conclusions, this summary equates paper "quality" with number of citations. High numbers of citations do not mean high quality, and is very field-dependent.
Quality can only be assessed by people reading the paper.
Every experiment which ends in a big bang is a good experiment.
People tend to cite papers from higher ranked journals more. In addition, said journals are higher ranked by search engines so the virtuous cycle continues. Thus this result is bogus.
The right way to do this study is to do a controlled study. Fortunately, this has been done in the recent NIPS conference (note that in CS conferences are more important than journals). See http://mrtz.org/blog/the-nips-experiment/ for details. Essentially, the noise is *huge*.
An excerpt:
"Relative to what people expected, 57% is actually closer to a purely random committee, which would only disagree on 77.5% of the accepted papers on average:"
University level:
I Just finished my PhD at Imperial College, a world leading institute, and all they care about is incremental research that industry will fund. New ideas just get thrown away until another university does the ground work and the IC jumps in with bigger wallets and then takes it on/steals it.
UK Government funding:
If you have a new or interesting approach forget about getting grant funding, you only get money in the UK if the work has already been proven to be successful. Quite literally the funding peer review of your grant can be rejected because you don't have the end answer (with a high degree of certainty) the research would give.
Original ideology of peer review is great, what is being practiced today (at least in the UK) is broken.
This is yet another statistical study that itself has flaws.
1.What was the selection process for the studies. The phrase "All 14 of the most highly cited papers in the study" implies that there were papers not in the study. Possible selection bias?
2. They do not go into why the 14 papers were cited so much and if any further research or refinement of the papers were done before they were accepted by other journals. Surface analysis of numbers can be manipulated to say anything.
3. They also say that it might be better to not have a review and just publish everything. This just means that everyone who reads the papers has to do the review. That is not practical. There are many papers that should not be published due to shoddy practices or malfeasance. Instead of trowing out the whole system how about looking at why the 14 papers were rejected and modifying the system accordingly.
4. The article does not give access to the study so we can review it. It is behind a paywall.
I would say the finding confirms that unconventional research that falls outside the established lines of thought may be more prone to rejection from top journals.
It's typical human politics and ideology at work. What would you expect from a large group of people, all with vested interests?
“He’s not deformed, he’s just drunk!”
I frequently had papers rejected as "not new" without citation and then accepted elsewhere where they told me on request that they checked carefully and found the content was indeed new and interesting. My take is that this may also quite often be "reviewers" that are envious or trying to compete by putting others down, not only ones that are incompetent. Fortunately, I had nothing stolen by reviewers (after they rejected it) but I know people that had this happen to them. As the peer-review system is set up at this time, the best and the brightest need longer to get their things published, need much longer to get PhDs and have significantly lower chances at an academic career as a result. Instead those that publish a lot of shallow and/or meaningless incremental research get all the professorships, while barely qualifying as scientists. The most destructive result of this is that many fields have little or no real advancement going on as new ideas are actively squashed. After all, new ideas would show how abysmally bad at science the current position-holders are. But this is not new. Cranks like Newton had research by others squashed or suppressed, because it was better than his stuff. This is also the reason most scientific discoveries take about one research-career-length to go into actual use. The thing is that the perpetrators of the status-quo have to retire before their misconceptions and mediocre approaches can be replaced. Exceptionally stupid, but all too human.
Personally, I am now in an interesting industrial job, and I still do the occasional paper as a hobby. But I would strongly advise anybody bright against trying for an academic career. Wanting to do research right reliably ensures that you will not be able to do an academic career at all. The core ingredients for a scientific career are mediocrity, absence of brilliance, hard work and a lot of political maneuvering. Oh, and you must not care about doing good science!
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
"unconventional... may be more prone to rejection"
Isn't that the definition of 'unconventional'?
Like many things in life, it's a popularity contest first, and a meritocracy second (at best).
"Unconventional methodology" is not.
Papers that don't use sufficiently rigorous methods should be rejected, regardless of their conclusions - even if those conclusions eventually turn out to be right. It's the only way to have any confidence about the research. If the authors are so sure of their results, they should do them more carefully, and submit again.
Far too often, rejections are taken as evidence of cronyism or groupthink (usually by those whose beliefs are contradicted by established science), when it's simply obvious flaws in methodology. When your methods are bulletproof, only then you can expect with confidence to pass review.
Why would anyone engrave "Elbereth"?
Maybe those 14 articles were cited more because they weren't buried under the paywalls imposed by the three "elite" journals? Scientists could actually get their eyes on these articles without paying a steep subscription or per unit cost?
Maybe the elite journals actually hinder the exchange of information and ideas that science needs to move forward? Nah! Can't be!
Left MS Windows for Linux Mint and never looked back!
Vote for Bernie in 2016!
> Considering the papers were eventually published anyway
That seems to be a clear cut case of biased sampling. What you heard about is all there is?
http://en.wikipedia.org/wiki/S...