Does Journal Peer Review Miss Best and Brightest?

sciencehabit writes: A study published today indicates that the scientific peer review system does a reasonable job of predicting the eventual interest in most papers, but it may fail when it comes to identifying really game-changing research. Papers that were accepted outright by one of the three elite journals tended to garner more citations than papers that were rejected and then published elsewhere (abstract). And papers that were rejected went on to receive fewer citations than papers that were approved by an editor. But there is a serious chink in the armor: All 14 of the most highly cited papers in the study were rejected by the three elite journals, and 12 of those were bounced before they could reach peer review. The finding suggests that unconventional research that falls outside the established lines of thought may be more prone to rejection from top journals.

Comments

Uhm, this place is peer reviewed...

[The New Guy 2.0]

For those of you just joining us, peers given mod points hand them out to review comments posted here. CmdrTaco's site has seen a lot of controversies with this system in the earlier days, and M2 was invented to review the mod point decisions. Lots of discussion has been sorted by this system, and the crap found on other servers has been eliminated.

Re:Uhm, this place is peer reviewed...

[omfgnosis]

And some of the most interesting, insightful, informative and funny comments go on forever underrated, and mostly unseen. This is caused in part by an understandable but evidently imperfect bias in Slashdot's design, where incumbent posts (posted earliest, posted by trusted users) are given greater visibility.

Like peer review in journals, it is possible that mostly-positive solutions can have negative consequences as well.

Obligatory Einstein Quote

[theshowmecanuck]

We cannot solve our problems with the same thinking we used when we created them.
-- Albert Einstein

Full disclosure: I found this at this web site.

Not a measure of quality

[toQDuj]

While I don't disagree with the conclusions, this summary equates paper "quality" with number of citations. High numbers of citations do not mean high quality, and is very field-dependent.
Quality can only be assessed by people reading the paper.

Correlation is not causation.

[Anonymous Coward]

People tend to cite papers from higher ranked journals more. In addition, said journals are higher ranked by search engines so the virtuous cycle continues. Thus this result is bogus.

The right way to do this study is to do a controlled study. Fortunately, this has been done in the recent NIPS conference (note that in CS conferences are more important than journals). See http://mrtz.org/blog/the-nips-experiment/ for details. Essentially, the noise is *huge*.

An excerpt:
"Relative to what people expected, 57% is actually closer to a purely random committee, which would only disagree on 77.5% of the accepted papers on average:"

Paywalls?

[Irate Engineer]

Maybe those 14 articles were cited more because they weren't buried under the paywalls imposed by the three "elite" journals? Scientists could actually get their eyes on these articles without paying a steep subscription or per unit cost?

Maybe the elite journals actually hinder the exchange of information and ideas that science needs to move forward? Nah! Can't be!

Survivorship bias!

[mha]

> Considering the papers were eventually published anyway

That seems to be a clear cut case of biased sampling. What you heard about is all there is?

http://en.wikipedia.org/wiki/S...