Does Journal Peer Review Miss Best and Brightest?

sciencehabit writes: A study published today indicates that the scientific peer review system does a reasonable job of predicting the eventual interest in most papers, but it may fail when it comes to identifying really game-changing research. Papers that were accepted outright by one of the three elite journals tended to garner more citations than papers that were rejected and then published elsewhere (abstract). And papers that were rejected went on to receive fewer citations than papers that were approved by an editor. But there is a serious chink in the armor: All 14 of the most highly cited papers in the study were rejected by the three elite journals, and 12 of those were bounced before they could reach peer review. The finding suggests that unconventional research that falls outside the established lines of thought may be more prone to rejection from top journals.

Comments

Uhm, this place is peer reviewed...

[The New Guy 2.0]

For those of you just joining us, peers given mod points hand them out to review comments posted here. CmdrTaco's site has seen a lot of controversies with this system in the earlier days, and M2 was invented to review the mod point decisions. Lots of discussion has been sorted by this system, and the crap found on other servers has been eliminated.

Re:Uhm, this place is peer reviewed...

[omfgnosis]

And some of the most interesting, insightful, informative and funny comments go on forever underrated, and mostly unseen. This is caused in part by an understandable but evidently imperfect bias in Slashdot's design, where incumbent posts (posted earliest, posted by trusted users) are given greater visibility.

Like peer review in journals, it is possible that mostly-positive solutions can have negative consequences as well.

Obligatory Einstein Quote

[theshowmecanuck]

We cannot solve our problems with the same thinking we used when we created them.
-- Albert Einstein

Full disclosure: I found this at this web site.

Not a measure of quality

[toQDuj]

While I don't disagree with the conclusions, this summary equates paper "quality" with number of citations. High numbers of citations do not mean high quality, and is very field-dependent.
Quality can only be assessed by people reading the paper.

Re:Not a measure of quality

[drooling-dog]

Also, a lot of highly-cited papers are methodological in nature, and the "Big 3" tend not to publish many of those.

Correlation is not causation.

[Anonymous Coward]

People tend to cite papers from higher ranked journals more. In addition, said journals are higher ranked by search engines so the virtuous cycle continues. Thus this result is bogus.

The right way to do this study is to do a controlled study. Fortunately, this has been done in the recent NIPS conference (note that in CS conferences are more important than journals). See http://mrtz.org/blog/the-nips-experiment/ for details. Essentially, the noise is *huge*.

An excerpt:
"Relative to what people expected, 57% is actually closer to a purely random committee, which would only disagree on 77.5% of the accepted papers on average:"

This is not a suprise

[Anonymous Coward]

University level:
I Just finished my PhD at Imperial College, a world leading institute, and all they care about is incremental research that industry will fund. New ideas just get thrown away until another university does the ground work and the IC jumps in with bigger wallets and then takes it on/steals it.

UK Government funding:
If you have a new or interesting approach forget about getting grant funding, you only get money in the UK if the work has already been proven to be successful. Quite literally the funding peer review of your grant can be rejected because you don't have the end answer (with a high degree of certainty) the research would give.

Original ideology of peer review is great, what is being practiced today (at least in the UK) is broken.

Re:This is not a suprise

[excelsior_gr]

This happens in Germany as well. When we applied for a government grant we had to present a detailed project plan and describe the "deliverables" in ridiculous detail. The people in the review committee weren't idiots, they knew that the plan was bullocks, but you had to include it anyway. Back then I attributed the whole thing to the german obsession with planning.

The finding suggests..?

[fustakrakich]

I would say the finding confirms that unconventional research that falls outside the established lines of thought may be more prone to rejection from top journals.

It's typical human politics and ideology at work. What would you expect from a large group of people, all with vested interests?

Paywalls?

[Irate Engineer]

Maybe those 14 articles were cited more because they weren't buried under the paywalls imposed by the three "elite" journals? Scientists could actually get their eyes on these articles without paying a steep subscription or per unit cost?

Maybe the elite journals actually hinder the exchange of information and ideas that science needs to move forward? Nah! Can't be!

Survivorship bias!

[mha]

> Considering the papers were eventually published anyway

That seems to be a clear cut case of biased sampling. What you heard about is all there is?

http://en.wikipedia.org/wiki/S...